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PREFACE 


his handbook serves as a guide to concepts and methods applied in 

the analysis of policy. I have developed the general approach and 
many of the specific suggestions over thirty-five years of teaching policy 
analysis workshops to first- and second-year graduate students at the 
Richard and Rhoda Goldman School of Public Policy, University of 
California, Berkeley. In the handbook’s earliest incarnation, the ideas 
took form slowly and were conveyed to students in lectures. But because 
my faculty colleagues and I systematically overloaded our students with 
work, they would sometimes skip a lecture—and thus miss out on ideas 
that I regarded as essential. I determined that if I were to create a handout 
for the students, at least I would have discharged my responsibility, and 
it would be up to the students to retrieve the ideas they had missed. Over 
the years, as the handout grew, it was disseminated informally to col- 
leagues at other universities and was posted on the Web site of the 
Electronic Hallway, based at the University of Washington. This book is 
the outgrowth of these previous compilations and the product of many 
years of experience. 

The presumed user is a beginning practitioner preparing to undertake a 
policy analysis, such as one of our master’s students at the Goldman School. 
But I have found this handbook useful at both ends of the spectrum—in 
teaching undergraduate introduction to public policy courses as well as 
executive education groups. 
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The handbook assumes a familiarity with basic economic concepts, 
including those having to do with market failures (including market 
imperfections). It is not meant to stand alone but should be used in con- 
junction with other sources, including some of the best textbooks in 
policy analysis, which are cited often to amplify points in this handbook: 
Weimer and Vining (2004); Stokey and Zeckhauser (1978); Behn and 
Vaupel (1982); Friedman (2002); MacRae and Whittington (1997); 
Gupta (2010); and Morgan and Henrion (1990). 

This new edition of A Practical Guide for Policy Analysis clarifies some 
of the exposition, particularly with regard to “design problems” and the 
choice of a “base case.” It also substitutes a new set of environmental 
problems in Table I and the surrounding discussion that is more up-to- 
date and more interesting than the previous one. 
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wish to acknowledge the patience and helpful response of the students 

and friends who have used this handbook, especially those who withstood 
its earlier versions. Special thanks are due Robert Behn, Joy Bonaguro, 
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however. Sasha was in my undergraduate Public Policy 101 class in 1991. 
An unusually gifted and entrepreneurial fellow, he once accosted me with 
this announcement: “Professor Bardach, these handouts you give us are 
outstanding. You should publish a book. When I graduate, Pm creating 
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my own publishing house, and your book is the first ’m putting out 
I said, “Sasha, you are surely mad. It’s fine by me, but I am not going to 
be responsible for your financial losses. You are on your own.” Sasha did 
exactly as promised. Alas, his publishing venture, Berkeley Academic 
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editions. I hope Iam now making sufficient amends. 


INTRODUCTION 


poio analysis is a social and political activity. True, analysts take moral 
and intellectual responsibility for the quality of their policy-analytic 
work. But policy analysis goes beyond personal decision making. First, 
the subject matter concerns the lives and well-being of large numbers of 
their fellow citizens. Second, the process and results of policy analysis 
usually involve other professionals and interested parties: it is often done 
in teams or officewide settings; the immediate consumer is a “client” of 
some sort, such as a hierarchical superior; and the ultimate audience will 
include diverse subgroups of politically attuned supporters and oppo- 
nents of the analysts work. All of these facts condition the nature of 
policy analysis and have a bearing on the nature of what is meant by 
“quality work.” 

A policy analyst can work in any number of positions. Once upon a 
time, the term implied a rather wonkish individual who worked in a large 
government bureaucracy, serving up very technical projections of the 
possible impacts of one or more policy alternatives to some undersecre- 
tary of planning. No longer. Today’s policy analysts help in planning, 
budgeting, program evaluation, program design, program management, 
public relations, and other functions. They work alone, in teams, and in 
loose networks that cut across organizations. They work in the public, 
nonprofit, and for-profit spheres. Although their work is ideally distin- 
guished by transparency of method and interpretation, the analysts 
themselves may explicitly bring to their jobs the values and passions of 
advocacy groups as well as the technical expertise of “neutral” civil 
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servants. The professional networks in which they work may contain—in 
most cases, do contain—colleagues drawn from law, engineering, 
accounting, and so on, and in those settings the policy-analytic point of 
view has to struggle for the right to counter—or, better yet, synthesize— 
the viewpoints of these other professionals. Although policy-analytic 
work products typically involve written reports, they may also include 
briefings, slide presentations, magazine articles, and television inter- 
views. The recipients of these products may be broad and diffuse audi- 
ences as well as narrowly construed paying clients or employers. 

The advice in this handbook is directed both to policy analysts in 
practice and to students and others who, for whatever reasons, are 
attempting to look at the world through the eyes of a practitioner. 


THE EIGHTFOLD PATH 


Policy analysis is more art than science. It draws on intuition as much as 
on method. Nevertheless, given the choice between advice that imposes 
too much structure on the problem-solving process and advice that 
offers too little, most beginning practitioners quite reasonably prefer too 
much. I have therefore developed the following approach, which I call the 
Eightfold Path: 


Define the Problem 
Assemble Some Evidence 
Construct the Alternatives 
Select the Criteria 

Project the Outcomes 
Confront the Trade-offs 
Decide! 

Tell Your Story 


These steps are not necessarily taken in precisely this order, nor are all of 
them necessarily significant in every problem. However, an effort to define 
the problem is usually the right starting place, and telling the story is almost 
inevitably the ending point. Constructing alternatives and selecting criteria 
for evaluating them must surely come toward the beginning of the process. 
Assembling some evidence is actually a step that recurs throughout the 
entire process, and it applies particularly to efforts to define the problem 
and to project the outcomes of the alternatives being considered. 
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The primary utility of this structured approach is that it reminds you 
of important tasks and choices that otherwise might slip your mind; its 
primary drawback is that, taken by itself, it can be mechanistic. 


The Problem-Solving Process 


The problem-solving process—being a process of trial and error—is 
iterative, so you usually must repeat each of these steps, sometimes more 
than once. 

The spirit in which you take any one of these steps, especially in the 
earliest phases of your project, should be highly tentative. As you move 
through the problem-solving process, you will probably keep changing 
your problem definition, as well as your menu of alternatives, your set of 
evaluative criteria, and your sense of what evidence bears on the problem. 
With each successive iteration you will become a bit more confident that 
you are on the right track, that you are focusing on the right question, and 
so on. This can be a frustrating process, but it can also be rewarding—if 
you learn to enjoy the challenges of search, discovery, and invention. 


Some of the guidelines are practical, but most are conceptual. Most of 
the concepts used will seem obvious, but there are exceptions. First, tech- 
nical terms are sometimes employed. Second, some commonsense terms 
may be used in a special way that strips them of certain connotations and 
perhaps imports others. For the most part, all these concepts will become 
intelligible through experience and practice. 


The concepts come embedded in concrete particulars. In real life, 
policy problems appear as a confusing welter of details: personalities, 
interest groups, rhetorical demands, budget figures, legal rules and inter- 
pretations, bureaucratic routines, citizen attitudes, and so on. Yet the 
concepts described in this handbook are formulated in the abstract. You 
therefore need to learn to “see” the analytic concepts in the concrete 
manifestations of everyday life. 


Caution: sometimes, some steps are already determined. Suppose 
your client says, “We need an extra million dollars to run this program in 
the next budget year: find it.” Does the Eightfold Path apply to this 
“analysis”? In a limited way. The client has already defined the problem 
and narrowed the relevant criteria very tightly. There won't be much 


xviii INTRODUCTION 


creative scope for you when it comes to those steps. But all the other steps 
are likely to be relevant. 

This challenge to “find it” is a simplified version of a more complex 
challenge—“Design it,” as in “Figure out [that is, ‘design’] a way to pro- 
tect this subway system from terrorist attack.” Here, too, the problem 
definition step has already been settled by the client, though the other 
steps are likely to get the creative juices flowing. Ideas for dealing with 
design problems in general are introduced in the section headed “Step 
Three: Construct the Alternatives.” 


Your Final Product 


So what will your final product look like? Here is a very rough sketch of 
a typical written policy-analytic report: 


e Ina coherent narrative style you describe some problem that needs 
to be mitigated or solved. 

e You lay out a few alternative courses of action that might be taken. 

e To each course of action you attach a set of projected outcomes that 
you think your client or audience would care about, suggesting the 
evidentiary grounds for your projections. 

e If no alternative dominates all other alternatives with respect to all 
the evaluative criteria of interest, you indicate the nature and mag- 
nitude of the trade-offs implicit in different policy choices. 

e Depending on the client’s expectations, you may state your own 
recommendation as to which alternative should be chosen. 


The Spirit of the Eightfold Path 


The spirit of the Eightfold Path is, I hope, economizing and uplifting. 
Analyzing public policy problems is a complex activity. It is easy to get 
lost, to waste a lot of time, to become demoralized. Other manuals and 
textbooks in policy analysis are primarily concerned that you get the 
analysis “right,” in some sense. This one should help in that respect, too. 
But, in addition, I hope that this handbook will help you to get it done 
with reasonable efficiency—and with a minimum of anxious confusion. 

Finally, just as policy analysis originates in politics, so it concludes 
in politics. Political life has two sides: channeling conflict and building 
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community. Policy analysis serves both sides. It channels conflict by 
showing that some arguments, and their proponents, are in some sense 
superior to others and deserve to win out. But it helps to build commu- 
nity by marking off potential common ground as well. This common 
ground is defined by the rules and conventions of rational discourse— 
where opponents may employ analytical procedures to resolve disagree- 
ments, or where they may discover that at least some seemingly irreduc- 
ible values conflicts can be recast as dry-as-dust technical disagreements 
over how much higher a probability Policy A has than Policy B for 
mitigating Problem P. 


OVERVIEW OF THE BOOK 


This book is a compilation of many component parts. The primary com- 
ponent is Part I, describing the Eightfold Path and recommending heu- 
ristics to help you negotiate it. 

Part II focuses on one particular step in the Eightfold Path: assembling 
evidence. It first appeared thirty-five years ago as a journal article, but I 
have since modified it and tried to integrate it better into the overall book 
in terms of both style and content. I include it because its objective is, I 
think, unique among the many prescriptive works in the social sciences 
and in journalism about data gathering and interpretation: it is, above all, 
concerned with using the researcher’s time and energy efficiently. 

Part III also addresses a specialized topic in policy analysis not dealt 
with in other works: making use of ideas—and specimens of “smart 
practices” —that are to be found in other sites. Imitation and adaptation 
are standard routes to progress (albeit occasionally, to regress) in other 
areas of life, so why not in public policy? 

In previous editions of this book, the third appendix offered a sum- 
mary of “semantic tips.” It may not surprise readers to learn that seman- 
tic pitfalls abound on the Eightfold Path of policy analysis, but it will 
surely be more surprising that there are many semantic tricks to help 
policy analysis along. Tips about these pitfalls and the tricks that can help 
to bridge them appear throughout this volume, and these practical rec- 
ommendations—no longer collected in a separate appendix—are now 
highlighted in context by means of a new semantic tips icon that appears 
at the beginning of each such discussion in Parts I, II, and III. 
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Appendix A contains the preface and summary sections of a lengthy 
study by researchers at the RAND Corporation on the relative worth of 
mandatory minimum sentences for drug dealers. I include this material 
because many students and fellow teachers have voiced the wish for a 
specimen of real, high-quality policy analysis, along with some commen- 
tary to highlight its most effective characteristics. Institutionally, RAND 
is the oldest policy research organization in the country, if not the world. 
It has a deserved reputation for excellence. It also has a tradition of doing 
cost-effectiveness analysis, beginning with work for the military and, in 
the past two or three decades, branching into domestic policy. It is there- 
fore particularly fitting to make use of a specimen produced by RAND. 

Why this particular specimen? Not because its conclusions are neces- 
sarily correct; I do not know enough about the drug policy field to have 
an opinion. I do know that these are highly respected researchers in the 
field and that the work selected here highlights the fifth, “Project the 
Outcomes,” step in policy analysis—which I pronounce “the hardest step,” 
mostly because of the uncertainties involved in projecting the future. The 
chosen specimen wrestles in an interesting way with such uncertainties. It 
also takes a creative approach to defining alternatives. And it illustrates 
clearly how to integrate strictly analytical work into a larger policy discus- 
sion rife with the value disagreements that are inevitably present in a 
democratic, pluralistic society. 

Appendix B, “Things Governments Do,” is a condensed survey of 
eleven types of governmental instruments for intervening in society. This 
new edition also offers both a new Appendix C, “Understanding Public 
and Nonprofit Institutions: Asking the Right Questions,” and an Appendix 
D, “Strategic Advice on the Dynamics of Political Support.” 

I have tried to keep the style simple and the text short. But the topics 
covered are numerous and complicated. The result is that the book is in 
some respects very dense. My students tell me that the book should be 
treated not just as a quick and pleasurable read, which of course it is, but 
as a reference volume to be experienced again and again for its delicious 
subtleties. No doubt they are right. 
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THE EIGHTFOLD PATH 


he analytic work in problem solving generally proceeds in a certain 

direction, from defining the problem at the beginning all the way to 
making a decision and explaining it at the end. But remember, this is a pro- 
cess much given to reconsidering, reviewing, changing your mind—in other 
words, retracing your steps on the path before starting out once more. Also, 
in some cases, the client or, perhaps, the political situation has already nar- 
rowed and focused the analytic task to such a degree that you need not even 
bother thinking through some of the steps. The exposition that follows lays 
out a generic process that must be adapted to particular contexts. 


STEP ONE: DEFINE THE PROBLEM 


Your first problem definition is a crucial step: it gives you both a reason 
for doing all the work necessary to complete the project and a sense of 
direction for your evidence-gathering activity. And in the last phases of 
the policy analysis, your final problem definition will probably help you 
structure how you tell your story. 


Think of Deficit and Excess 
A] i often—but not always—helps to think in terms of deficit 


and excess. For instance: 


e “There are too many homeless people in the United States.” 
e “The demand for agricultural water is growing faster than our ability 
to supply it at an acceptable financial and environmental cost.” 
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e “California’s population of school-age children is growing by 
140,000 per year, and our ability to develop the physical facilities in 
which to educate them is not growing nearly as fast.” 


It often helps to include the word too in the definition—as in “too big,” 
“too small,” “growing too slowly,” “growing too fast.” These last two 
phrases (about “growing”) remind us that problems deserving our atten- 
tion don’t necessarily exist today but are (at least potentially) in prospect 
for the future, whether near or distant. 

However, it does not help to think in terms of deficit and excess when 
your problem is an already well-structured decision choice—for example, 
“Dump the dredging spoils either in the Bay or somewhere out in the 
Pacific Ocean.” Nor does it help if your challenge is to invent any way to 
accomplish some defined objective—for example, “Find some grant 
funds to close the anticipated gap between revenues and expenditures.” 
These decision- and invention-type challenges are problems for the pol- 
icy analyst but are not the substantive sort of problems I am addressing 
in this section. 


Make the Definition Evaluative 


Remember that the idea of a “problem” usually means that people think 
there is something wrong with the world, but note that wrong is a very 
debatable term. Not everyone will agree that the facts you (or others) 
have defined as a problem really do constitute a problem, for each person 
may apply a different evaluative framework to these facts. Unfortunately, 
there are no obvious or accepted ways to resolve philosophical differ- 
ences of this type. 

A common philosophical as well as practical question is this: “What 
private troubles warrant definition as public problems and thereby 
legitimately raise claims for amelioration by public resources?” It is usu- 
ally helpful to view the situation through the “market failure” lens 
(Weimer and Vining 2004, chap. 5).! In its simplest formulation, market 


1. For an analysis of most traditional market failures in transaction cost terms, see 
Zerbe and McCurdy (1999), which also emphasizes the rich variety of interventions 
besides those undertaken by government to remedy traditionally conceived “market 
failures.” 
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failure occurs when the technical properties of a good or service have 
one of the following effects: 


e Making it hard to collect payment from all the potential beneficiaries— 
for instance, the large number of people who profit, albeit indi- 
rectly, from advances in basic science 

e Making it hard to collect from the beneficiaries of consumption the 
true economic cost of making use of the good or service—such as 
the fresh air that vehicle owners use as a sink for their auto emissions 

e Making it hard for consumers (and sometimes suppliers) to know 
the true qualities of the good or service they are acquiring—for 
instance, many repair-type services, including those performed by 
physicians as well as those performed by auto mechanics 

e Making the cost of producing the marginal unit lower than the 
average cost within the relevant range of demand—such as a maga- 
zine article distributed via the Internet 


It is impossible to overestimate the importance of this point. In most— 
though not all—situations in which no actual market failures can be iden- 
tified, people’s private troubles cannot typically be ameliorated by even the 
most well-intentioned governmental interventions. Even when some ame- 
lioration is possible, there are usually many adverse side effects. In some 
cases, it may nevertheless be worthwhile to pay the price of these side 
effects, but such calculations must be done carefully and scrupulously. 

Besides market failures, the main situations in which private troubles 
can warrant definition as public problems are these: 


e Breakdowns of systems, such as family relationships, that occur 
largely outside markets 

e Low living standards that arise precisely because markets do func- 
tion well and do not reward individuals very generously if they lack 
marketable talents or skills 

e The existence of discrimination against racial and other minorities 

e The failure of government to function well in areas in which it is 
traditionally expected to act effectively (e.g., in providing public 
schools) 
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Using issue rhetoric. Usually, the raw material for the evaluative aspect of 
your initial problem definition comes from your client and derives from 
the ordinary language of debate and discussion in the client’s political 
environment—language that I call generically “issue rhetoric.” Such 
rhetoric may be narrowly confined to a seemingly technical problem or 
broadly located in a controversy of wide social interest. In either case, you 
have to get beyond the rhetoric to define a problem that is analytically 
manageable and that makes sense in light of the political and institu- 
tional means available for mitigating it. 

Use the raw material of issue rhetoric with care. It often points to 
some condition of the world that people don’t like or consider “bad” 
in some sense, such as “teenage pregnancy,” “media violence,” or “glob- 
al warming.” These evaluations do not necessarily need to be taken at 
face value. You will sometimes wish to explore the philosophical and 
empirical grounds on which you, your client, or others in your even- 
tual audience should or should not consider the alleged condition 
“bad.” Furthermore, issue rhetoric may point to some alleged—but not 
necessarily real—cause of the troubling condition, such as “welfare” or 
“human wastefulness.” 

Issue rhetoric often has a partisan or ideological flavor. Although 
Americans cluster toward a mixed-ideology and pragmatic center, issue 
rhetoric is created by the more passionate and often more articulate indi- 
viduals who settle closer to the extremes. The great ideological divide in 
most developed democracies concerns the role of government assistance 
and regulation in solving problems relative to reliance on self, kin, and 
neighbors. Self-reliance is generally presumed to be the ideal, but this is 
a rebuttable presumption. “Liberal” issue rhetoric typically offers many 
rebuttals, usually involving distrust of “the market,” but only some of 
these rebuttals are grounded in realistic understanding of how markets 
do and do not work. “Conservative” issue rhetoric sometimes offers 
thoughtless defenses of “the market” but can also fall silent when favored 
business interests seek protectionist legislation. Because government as 
an institution is the chief alternative to private and community problem- 
solving, liberals and conservatives alike ideologize the question of just 
how competent and trustworthy it is. Selective perception abounds on 
both sides of this argument. 
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Generalities originating in issue rhetoric only sometimes suffice to 
settle concrete issues of policy choice and policy design, although eco- 
nomic theories of market failures and imperfections can often tell us 
when not to rely on the market, and public choice theories of government 
failure can often tell us when not to rely on the government (Weimer and 
Vining 2004; Glazer and Rothenberg 2001). Policy analysis typically 
bridges all political ideologies by reliance on the normative standard of 
“maximizing welfare” and on social science theorizing about the com- 
parative advantages of different institutions for different purposes. Thus 
you want not simply to echo the issue rhetoric in your problem definition, 
but to use it as raw material for a provisional problem definition that you 
hope will prove analytically useful. 

Note also that some issue labels may signify more than one problem. 
Depending on the audience, for example, “teenage pregnancy” may con- 
note any or all of the following conditions: sexual immorality, the blight- 
ing of young people’s and their children’s life chances, exploitation of 
taxpayers, and social disintegration. Usually you will want to determine 
a primary problem focus, to ensure that the analysis does not get out of 
hand. But if the problems aren’t too complicated, you may feel willing to 
define more than one. 


Quantify If Possible 


Your problem definition should, insofar as possible, include a quantitative 
feature. Assertions of deficit or excess should come with magnitudes attached. 
How big is “too big”? How small is “too small”? How about “too slowly” or 
“too fast”? With regard to homelessness, how many homeless people are 
there in the United States? Or in the case of agricultural water, how many 
acre-feet of water are used now, and how does that amount compare with the 
demand in some specified future year (given certain assumptions about 
water pricing)? Exactly what is “our ability to develop physical facilities for 
water storage,” and how do we expect it to grow, or shrink, over time? 

If necessary, gather information to help you calibrate the relevant mag- 
nitudes. (See the discussion under “Step Two: Assemble Some Evidence.) 

In many or most cases, you will have to estimate—or, more likely, 
“guesstimate’—the magnitudes in question. Sometimes you should 
furnish a range as well as a point estimate of magnitudes—for example, 
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“Our best guess of the number of homeless persons in families is 
250,000, although the truth could lie between 100,000 and 400,000.” 

Even if you cannot come up with good numbers yourself, qualita- 
tively defining a metric that might be used to quantify the problem helps 
you make your problem definition more behavioral and concrete. It is 
better to say, “Too many people with annual incomes over $60,000 are 
living in subsidized apartments,” than simply, “Too many relatively well- 
off people are taking advantage of low-rent public housing.” The $60,000 
value provides desirable texture and information about a threshold num- 
ber that will serve in the promised analysis. 


Diagnose Conditions That Cause Problems 


Some problematic conditions are not experienced as troublesome per se 
by citizens but are perceived by them, or by analysts working on their 
behalf, to be causes of trouble. It is sometimes useful to diagnose at least 
one alleged condition of this type and to define it as a problem to be 
mitigated or removed—as in, “One of the problems in the air pollution 
area is that states have not been willing to force motorists to keep their 
engines tuned up and their exhaust systems in proper order.” 


A] Note that this sort of problem definition is not merely 
descriptive but is also diagnostic. It implicitly asserts that some condition, 
which people may or may not find troubling on its own, is an important 
cause of some other condition that is indeed troubling. Problem defini- 
tions that pretend to such diagnostic power can be useful, but they can 
also be treacherous. Suppose, after all, that the causal diagnosis is mis- 
taken or misleading—for example, that states’ unwillingness to enforce 
engine maintenance routines is not in fact a very important cause of air 
pollution. Because the term definition in some contexts connotes legiti- 
mate arbitrariness (“Ill define justice to mean . . .”), the causal claims 
implicit in diagnostic problem definitions can easily escape needed scru- 
tiny. (See “Step Five: Project the Outcomes” for further discussion.) 


Risky Conditions: “The Odds” 


“The odds are too high that this nuclear reactor will suffer an accident in 
the next twenty-five years that will emit excessive radiation.” This sentence 
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does indicate a problem, but it is not something tangible, like “too many 
cases of asthma are being reported in this neighborhood.” It refers to risk 
and is stated in probabilistic language dealing with “the odds.” 


Semantic Tip Referring to “the odds” is a useful way to talk about any- 
thing that is uncertain in your analysis, not just the problem definition. 
It can also refer to the uncertain prospects of an alternative’s working out 
as planned, or the likelihood that a key political actor will remain in 
office in order to oversee policy implementation. It is an especially useful 
locution when talking about uncertainties that are particularly resistant 
to quantification, for example, “The odds are that the U.S. nuclear mod- 
ernization program is causing other countries to look more favorably on 
acquiring nuclear weaponry themselves.” 

The odds formulation can also be used for specifying criteria. For 
instance, one could say that one criterion is “Maximize the odds that the 
People’s Party will control the upper chamber following the next election” 
or “minimize the odds that teens in the catchment area of this program 
will reject it because it is not ‘cool”” 


Identify Latent Opportunities 


A special kind of problem is an opportunity missed. Is it not rather small- 
minded to think of policy analysis as devoted merely to the amelioration of 
problems? May policy analysis not rise above the tedious and uninspiring 
business of patching and fixing? Can we not aspire to a world in which we 
can identify opportunities to do creative—not to say wonderful—things? 
“Tf it ain't broke, don’t fix it” is a confining idea, and certainly policy analysts, 
policymakers, and public managers ought not to allow the “problem” focus 
to restrict the search for plausible opportunities. Unfortunately, the working 
agenda of most policy professionals is set by complaints, threats, worries, 
and troubles—often leaving little time or energy to think about improve- 
ments that no one has identified as needful. Still, if latent opportunities are 
really lying around, it would be a pity to ignore them. 

Where do we find opportunities for creative policy improvements that 
haven’t first been identified by complaints, threats, and so on? Little aca- 
demic or technical theory is available to answer this question. But Box I-1 
(p. 8) contains a list that is suggestive. 
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| BOX | Some Generic Opportunities for Social Improvement 
That Often Go Unnoticed 


Operations research strategies. By means of sequencing, timing, prioritiz- 
ing, matching, clustering, and other such rationalizing arrangements, it 
may be possible to use a fixed stock of resources to achieve higher produc- 
tivity than is possible otherwise. For instance, provided that traffic flow 
conditions are within certain parameters, high-occupancy-vehicle (HOV) 
lanes can maximize vehicle throughput in a fixed section of roadway. 


Cost-based pricing. Discrepancies between prices and real costs present an 
opportunity for enhancing social welfare by adjusting prices to better 
reflect reality. For instance, introducing congestion tolls, eliminating cross- 
subsidies for peak-period utilization of electricity, or removing rent con- 
trols would each bring prices more into line with real costs. 


By-products of personal aspirations. It is possible to structure new incen- 
tives or create new opportunities for personal advantage or satisfaction that 
can indirectly result in social benefit. For example, public-sector employers 
can offer to share the benefits of cost-reducing innovations with the 
employees who conceive them and implement them. 


Complementarity. Two or more activities can potentially be joined so that 
each may make the other more productive. For example, increased public 
works construction can combat unemployment. 


Input substitution. The world abounds in opportunities to substitute less 
costly inputs in a current production process while achieving roughly 
equivalent results. For instance, municipalities can hire lower-paid civilians 
to perform police clerical tasks rather than use expensive uniformed officers. 


Avoid Common Pitfalls in Problem Definition 


Problem definition is a step beset by at least two dangerous pitfalls. 


mantic Tip| Defining the solution into the “problem.” Your problem 
definition should not include an implicit solution introduced by seman- 
tic carelessness. Projected solutions must be evaluated empirically and 
not legitimated merely by definition. Therefore, keep the problem defini- 
tion stripped down to a mere description, and leave open where you will 
look for solutions. 
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Development. A sequence of activities or operations may be arranged to 
take advantage of a developmental process. For example, a welfare agency can 
assess clients for employability and vocational interest before, rather than 
after, sending them out to search for a job. 


Exchange. Unrealized possibilities for exchange can increase social 
value. Policymakers typically design policies to simulate market-like 
arrangements—for example, conducting pollution permit auctions, or 
reimbursing an agency for services it renders to another agency’s clients 
or customers. 


Multiple functions. A system can be designed so that one feature has the 
potential to perform two or more functions. For example, a tax adminis- 
trator can dramatize an enforcement case in such a way as both to deter 
potential violators and to reassure nonviolators that they are not being 
played for suckers because of their honesty. 


Nontraditional participants. Line-level employees of public agencies—as 
well as their customers, clients, or the parties whom they regulate—often 
have knowledge of potential program improvements that could usefully 
be incorporated into the agencies’ policies and operations. The IRS, for 
instance, has sought feedback from ordinary tax filers about how to improve 
federal tax forms. 


Underutilized capacity. Governments sometimes systematically under- 
utilize resources at their disposal. In many communities, school facilities 
are used for relatively limited purposes for only part of the day and for 
only part of the year—although school officials would be quick to warn 
that tapping this capacity without harming school functions is not 
always easy. 
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e Don’t say: “There is too little shelter for homeless families.” Inadver- 


tently implying that “more shelter” is the best solution may inhibit you 
from thinking about ways to prevent families from becoming home- 
less in the first place. Try instead: “Too many families are homeless.” 
Don’t say: “New schools are being built too slowly.” Simply assum- 
ing that “more schools” is the solution may inhibit you from think- 
ing about ways to use existing facilities more efficiently or even to 
try forms of “distance learning.” Try instead: “There are too many 
schoolchildren relative to the currently available classroom space.” 
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A tip-off that you're probably smuggling an implicit solution into the 
problem definition is to hear yourself saying, “Aha, but that’s not the real 
problem; the real problem is . . .” While there are better and worse ways 
to conceptualize a problem—or to solve a problem—it stretches ordinary 
usage too much to say that one problem could be “more [or less] real” 
than another. 


Accepting too easily the causal claims implicit in diagnostic problem 
definitions. I suggested earlier that conditions that cause problems may 
also be problems themselves. However, the causes must be real, not 
merely assumed. You have to evaluate the causal chain that goes from 
the situation itself to the bad effects it is alleged to cause, and to con- 
vince yourself that the causal relationship is real. For instance, for some 
people, cocaine use is not a problem in itself, but it may become a prob- 
lem if it leads to crime, poor health, family disintegration, and so on. 
But does it lead to these outcomes, and to what degree? The evidence 
on this question should be evaluated very carefully before you decide 
that it’s okay to work with a problem definition that sounds like “too 
much cocaine use.” 


Iterate 


Problem definition is a crucial step. Because it is hard to get it right, how- 
ever, you may take that same step again and again. Also, your empirical 
and conceptual understanding will evolve over the course of your ana- 
lytic work. For instance, you may start out thinking that the main prob- 
lem is “too many halfway houses for the mentally ill in our city” but end 
up concluding that the main problem is how badly some of them are 
managed.” As you begin to rule out alternative approaches to solving or 
mitigating your problem, you will probably want to sculpt the problem 
definition so that, in the end, you and the political system will have some 
chance of attacking the problem successfully. Finally, if you are working in 
an office or agency context, you will implicitly be negotiating a mutually 


2. This happened to a graduate student group at the Goldman School whose client 
was the Oakland Police Department. Members of the group struggled hard to escape 
the initial assumptions held by their client and eventually to refocus their work. 
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acceptable problem definition with your analyst colleagues and your hier- 


archical superiors.’ 


STEP TWO: ASSEMBLE SOME EVIDENCE 


All of your time doing a policy analysis is spent on two activities: think- 
ing (sometimes aloud and sometimes with others) and hustling data 
that can be turned into evidence. Of these two activities, thinking is 
generally the more important, but hustling data takes much more time: 
reading documents, hunting in libraries, poring over studies and statis- 
tics, interviewing people, traveling to interviews, waiting for appoint- 
ments, and so on. 

The real-world settings in which policy analysis is done rarely afford 
the time for a research effort that would please a careful academic 
researcher. In fact, time pressure is probably almost as dangerous an 
enemy of high-quality policy analysis as is politically motivated bias, if 
not more so. Therefore, economize on your data collection activities. The 
key to economizing is this: try to collect only those data that can be 
turned into “information” that, in turn, can be converted into “evidence” 
that has some bearing on your problem. 


EA] For the logically minded, here are some definitions: Data 
are facts—or, some might say, representations of facts—about the 
world. Data include all sorts of statistics but go well beyond statistics, 
too. Data also include, for instance, facts about an agency manager’s 
ability to deal constructively with the press. Information consists of data 
that have “meaning,” in the sense that they can help you sort the world 
into different logical or empirical categories. The prevalence of cigarette 
smoking in five different countries constitutes data, but these data 
become information when you decide it is interesting to array the coun- 
tries comparatively (e.g., from lowest to highest prevalence). Evidence is 


3. Some analysts also claim that it is simply not worthwhile to define as “problems” 
conditions that cannot be ameliorated: “Problems are better treated as opportunities 
for improvement; defined problems, as problems of choice between alternative 
means to realize a given opportunity. The process of problem definition would then 
be one of search, creation, and initial examination of ideas for solution until a prob- 
lem of choice is reached.” See Dery (1984, 27). 
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information that affects the existing beliefs of important people 
(including yourself) about significant features of the problem you are 
studying and how it might be solved or mitigated. Differential preva- 
lence of smoking, for instance, can become evidence bearing on 
hypotheses concerning different levels of concern about personal 
health across countries. 

You need evidence for three principal purposes, all of which are rele- 
vant to the goal of producing realistic projections of possible policy 
outcomes. One purpose is to assess the nature and extent of the 
problem(s) you are trying to define. A second is to assess the particular 
features of the concrete policy situation you are engaged in studying. For 
instance, you may need to know—or guess—about agency workloads, 
recent budget figures, demographic changes in a service area, the political 
ideology of the agency chief, the competency of the middle-level manag- 
ers in the agency, and the current attitudes of some other agency that 
nominally cooperates with yours on some problem. The third purpose is 
to assess policies that have been thought, by at least some people, to have 
worked effectively in situations apparently similar to your own, in other 
jurisdictions, perhaps, or at other times. (Sometimes these situations will 
have been evaluated statistically and sometimes not: see Part III, “Smart 
(Best) Practices’ Research: Understanding and Making Use of What Look 
Like Good Ideas from Somewhere Else”) 

Because each of these purposes becomes salient in different phases of 
the policy analysis process, the second step on the Eightfold Path, 
“Assemble Some Evidence,” will be taken more than once but with a dif- 
ferent focus each time. 


Think Before You Collect 


Thinking and collecting data are complementary activities: you can be a 
much more efficient collector of data if you think, and keep on thinking, 
about what you do and don’t need (or want) to know, and why. The 
principal—and exceedingly common—mistake made by beginners and 
veterans alike is to spend time collecting data that have little or no 
potential to be developed into evidence concerning anything you actu- 
ally care about. People often do this because running around collecting 
data looks and feels productive, whereas first-rate thinking is hard and 
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frustrating. Also, when they see you busily collecting data, the people 
paying for your work tend to be reassured that somehow they are getting 
their money’s worth. 


The value of evidence. Since most evidence is costly to produce, you must 
weigh its likely cost against its likely value. How is its likely value to be 
estimated? The answer may be cast in a decision-analytic framework 
(decision trees), though remember that the process of making a decision 
involves a great many elements prior to the moment of actual choice, such 
as defining a useful problem, thinking up better candidate solutions, and 
selecting a useful model. In general, the value of any piece of evidence 
depends on these factors: 


e The likelihood that it will cause you to substitute some better deci- 
sion for whatever decision you would have made without it (which 
might have been an “acceptable” decision in and of itself) 

e The likelihood that the substituted decision will, directly or indi- 
rectly, produce a better policy outcome than the outcome that would 
have been produced by the original decision 

e The magnitude of the difference in value between the likely-to-be- 
improved outcome and the original outcome 


The utility of an educated guess. It is surprising how well you can do in 
many cases by gathering no evidence at all but simply by sitting down 
and thinking something through and then making some serious edu- 
cated guesses. There is nothing shameful about acting on such guessti- 
mates and thereby conserving your data-collecting time and energies for 
answering questions for which good evidence is really necessary (see Part II, 
“Assembling Evidence”). 

A helpful check on your thinking, to avoid collecting useless data, is to 
ask yourself the following questions before embarking on some data col- 
lection venture: 


e “Suppose the data turn out to look like so-and-so as opposed to 
thus-and-such. What implication would that have for my under- 
standing of how to solve this problem?” 
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e “Compared to my best guess about how the data will look once I’ve 
got them, how different might they look if I actually took the trouble 
to get them?” 

e “How much is it worth to me to confirm the actual difference 
between what I can guess and what I can learn about the world by 
really getting the data?” 


It is this sort of critical attitude about the value of expensive data col- 
lection (especially ad hoc surveys!) that often leads good and experienced 
policy analysts to make do with back-of-the-envelope estimates. However, 
none of this reasoning is meant to be an excuse for shirking the job of 
getting good data—and sometimes a lot of it, at huge costs in time and 
money—when you've convinced yourself that the investment really will 
pay off. There’s an obvious and critical difference between justifiable and 
unjustifiable guesstimates. 


Review the Available Literature 


There hardly exists a problem on whose causes and solutions some aca- 
demic discipline or professional association is not doing research. It is 
easy to find journals and various professional publications disseminating 
research results, theories, case studies, the musings of experienced prac- 
titioners, and so on. The Internet brings much of this literature to your 
desktop. 

Advocacy organizations often publish a great deal of interesting work 
and may take special pains to disseminate their findings on the Internet. 
However, because advocacy-based analyses are not, in general, as reliable 
as more disinterested work, there is a danger of relying too much on such 
sources just because they are readily available. 


Survey “Best Practices” 


The chances are good that the problem you are studying is not unique, 
and that policymakers and public managers in other jurisdictions, per- 
haps not very different from the one you are studying, have already dealt 
with it in some fashion. See if you can track down some of these past 
solutions and extrapolate them to the situation you are studying. Bear in 
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mind, however, that the extrapolation process is complicated (see Part HI, 


ce 


Smart (Best) Practices’ Research”). 


Use Analogies 


Sometimes it pays to gather data about things that, on the surface, seem 
quite unlike the problem you are studying but, on a deeper level, show 
instructive similarities. For instance, your understanding of how a merit 
pay plan for compensating managers in the public sector might work 
could perhaps be improved by seeing how similar schemes work in the 
private sector. Or, if you are working on the problem of how a state can 
discipline, and perhaps disbar, incompetent attorneys, you might use- 
fully spend a good deal of your time learning about how the medical 
profession handles problems of physician incompetence. If you are work- 
ing on how to reduce neighborhoods’ resistance to accepting low-income 
housing projects, you could usefully look into the literature on commu- 
nity resistance to accepting solid-waste incinerators. 

As these examples suggest, some analogies are easier to perceive, and to 
make sense of, than others. It takes a little imagination to see an instruc- 
tive analogy and, occasionally, a little daring to try to convince others to 
recognize both its usefulness and its inevitable limitations. 


Start Early 


You are often dependent on the very busy schedules of other people 
whom you ask to furnish information or to make time for an interview. 
It is extremely important to submit requests for information—and 
especially for interviews—well in advance of when you want to have 
completed the data collection. (For a useful description of how to con- 
duct literature reviews, library searches, phone interviews, and personal 
interviews, see Weimer and Vining 2004, chap. 13; see also Part II, 
“Assembling Evidence.”) 


Touch Base, Gain Credibility, Broker Consensus 


The process of assembling evidence inevitably has a political as well as a 
purely analytic purpose. Sometimes it entails touching base with poten- 
tial critics of your work so that they will not be able to complain later that 
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you have ignored their perspectives. Conversely, by making yourself 
known to potential supporters of your work, you may be able to create a 
cadre of defenders. 

A more complex objective, where appropriate, might be to blend 
policy analysis with the process of improving a policy idea or decision 
during the course of implementation. (See the following discussion of 
“improvability” as a practical criterion.) This objective entails obtaining 
feedback from participants, usually in an iterative process, and sharing 
some of your own reactions with them. You thereby become more of a 
partner in the process than an outside observer and diagnostician. An 
even more complex and challenging role would be for you to become a 
particular type of “partner,” a facilitator and broker, whether by acting as 
a conduit from one person to another or by convening meetings and 
other gatherings. 


Free the Captive Mind 


In exchange for access to data and a ready-made worldview, researchers 
sometimes uncritically accept problem definitions and preferred solutions 
from kindly informants (not to mention from paying clients or employers). 
To counter such temptations, be sure to make contact with individuals 
or factions whom you would expect to disagree—the more sharply the 
better—with those informants. A time-saving, but only partial, substitute 
is to ask your kindly informants, “Who might object strongly to your point 
of view about this, and why might they do so?”4 


STEP THREE: CONSTRUCT THE ALTERNATIVES 


By alternatives I mean something like “policy options,” or “alternative 
courses of action,” or “alternative strategies of intervention to solve or 
mitigate the problem.” 


Beware a Linguistic Pitfall 


Eemantic Tip| Specifying alternatives does not necessarily signify that 
the policy options are mutually exclusive. Policy analysts use the term 


4. For an excellent discussion of this process, see Klitgaard, forthcoming. 
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alternative ambiguously: sometimes it means one choice that implies fore- 
going another, and sometimes it means simply one more policy action 
that might help to solve or mitigate a problem, perhaps in conjunction 
with other alternatives. Be aware of the ambiguity in other people’s usage, 
and in telling your story (see Step Eight), be sure that no such ambiguity 
enters your own usage. 

Sometimes you won't be entirely sure whether two alternatives are or 
are not mutually exclusive. For instance, although the mayor may have 
promised enough money to either fix potholes or provide homeless shel- 
ters (but not both), you may have made such a great case for both pro- 
grams that the mayor may decide to increase the budgetary allocation. 


Start Comprehensive, End Up Focused 


In the last stages of your analysis, you won’t want to be assessing more 
than two or three principal alternatives, but in the beginning, err on the 
side of comprehensiveness. Make a list of all the alternatives you might 
wish to consider in the course of your analysis. Later on, you will discard 
some obvious losers, combine others, and reorganize still others into a 
single “basic” alternative with one or more subsidiary “variants.” For your 
initial list, though, where should you turn for ideas? 

One starting point would be to note the alternatives that key political 
actors are actively proposing or seem to have on their minds. These may 
include prominent people’s pet ideas, institutions’ inventories of “off- 
the-shelf” proposals that simply await a window of opportunity, and 
prepackaged proposals that political ideologues are perennially advocat- 
ing. Then you could try to design alternatives that might prove to be 
superior to the alternatives currently being discussed by the key political 
actors. It’s good to brainstorm, to try to be creative—but don’t expect 
that you will necessarily produce much better ideas than those that other 
people have already advanced. 

One way to coax your creativity is to refer to the checklist in Appendix B, 
“Things Governments Do.” For each entry on the list, ask yourself: “Might 
it make sense to try some version of this generic strategy to help miti- 
gate this problem?” Because it is a comprehensive list, the answer with 
respect to any single strategy will usually be no. Going through the list 
systematically is worthwhile, however. Because the list is not very long, 
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with experience you will need to spend only a few minutes to decide 
whether any ideas there might be worth considering further. (See also the 
valuable discussion about generic policy instruments in Weimer and 
Vining 2004, chap. 10.) 


mantic Tip| Always include in your first approach to the problem the 
alternative “Let present trends (or ‘business-as-usual’) continue undis- 
turbed.” You need to do this because the world is full of naturally occur- 
ring, ongoing changes, some of which may mitigate, or worsen, the 
problem on which you are working. (Note that I am not characterizing 
this alternative as “Do nothing.” It is not possible to do nothing or to “not 
decide.” Most of the trends in motion will probably persist and alter the 
problem, whether for better or for worse.) 

To see if “natural” change will affect the scope of the problem, inspect 
its most common sources in the public policy environment: (1) political 
changes following elections, as well as changes induced by the prospect of 
having to contest an election; (2) changes in unemployment and inflation 
rates that accompany the business cycle; (3) the changing “tightness” or 
“looseness” of agency budgets caused by overall taxing and spending poli- 
cies; (4) demographic changes, such as population migration patterns and 
population “bulges” moving through certain age levels; and (5) changing 
technologies. In most cases, however, this “let-present-trends-continue” 
option will drop out of your final analysis. It follows that if you do your 
problem definition work well, you will end up with an important problem 
in your sights that in most cases can be mitigated to some degree by pur- 
posive action. 

Another frequently helpful alternative is “Learn more.” This can be 
done by using pilot studies, or by looking around for examples of “smart 
practices” elsewhere (see Part III), or by waiting for the future to get less 
murky, or perhaps by negotiating further with important players to 
ascertain what they might do under various contingencies. Don’t forget 
that there is a cost to waiting if, in the absence of further learning, you 
would have guessed “the right conclusion” anyway. Conversely, there is a 
cost to premature decision making or action if you are likely to make a 
consequential mistake that could be corrected by further learning. 
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Model the System in Which the Problem Is Located 


We often think about alternative approaches to the problem as possible 
interventions in the system that holds the problem in place or keeps it 
going. Logically, it is not necessary to model the causes of a problem in 
order to cure it—pharmaceutical manufacturers can testify that many of 
their successful products work by unknown causal routes on conditions 
whose causes are not at all understood. But a good causal model is often 
quite useful for suggesting possible “intervention points.” This is espe- 
cially true when the problem is embedded in a complex system of inter- 
acting forces, incentives, and constraints—which is usually the case. 
Consider, for instance, a system that produces “too much traffic conges- 
tion” at some choke point such as a bridge or a tunnel. A sketch of the 
relevant causal model would include the demand for travel along the 
relevant route, the available alternative modes of travel, the amount of 
roadway capacity, and the price to users of roadway capacity. An efficient 
and simple—but usually politically unpopular—intervention might be 
to increase the price to users so as to reflect the degree to which each user 
contributes to congestion and increased travel times. 

How self-conscious, elaborate, and rigorous should your causal model 
be? Many social scientists who devote themselves to policy analysis would 
hold, “The more so the better.” I say, “Yes, but . . . .” Self-consciousness is 
highly desirable. Elaborateness (or comprehensiveness—in this case a 
near synonym) is desirable because it decreases the risk of missing impor- 
tant causal connections, but it can blur the analytic focus and blunt cre- 
ativity in designing intervention strategies. Rigor is desirable if it prevents 
you from relying on unarticulated and false assumptions; its downside 
is that it may persuade you to exclude factors that are important—for 
instance, the personalities of certain actors—because you don’t know 
how to model their effect rigorously or because you have only hunches 
regarding the facts. 

Many models are best thought of as elaborations of a fundamental 
metaphor. They can be mathematically precise or verbal and evocative. 
Some commonly used metaphors that are the bases for models of par- 
ticular value in designing alternatives are discussed in the following 
sections. 
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Market models. The model of a market in which disaggregated suppliers 
exchange goods or services with disaggregated demanders can apply to 
unpriced goods and services. The main idea behind the market model is 
really equilibration through exchange. Hence, the market model can be 
applied to many phenomena other than the production and allocation of 
textbook goods such as widgets or apples. 

For instance, you might try to understand the flow of patients into a 
state mental hospital system in terms of supply and demand: there is a 
fixed short-run “supply” of available beds in state hospitals and a per- 
diem charge for each, and a complex “demand” for their use generated by 
police departments, county psychiatric emergency units, judges, mem- 
bers of the public, and so on. 

A standard intervention strategy for improving markets that are not 
working as well as they might is to find some way to raise or lower the 
prices faced by either suppliers or demanders. 


Production models. Unfortunately, little academic literature has exam- 
ined the operating logics of the common types of production systems 
found in public policy—such as command-and-control regulation, the 
provision of information, and all the other “Things Governments Do” 
that are described briefly in Appendix B. (However, see Weimer and 
Vining 2004, chap. 10, on “generic policies”; see also Salamon 2002.) In 
any case, the main concern in understanding production systems should 
be to identify the parameters whose values, when they move out of a 
certain range, make the systems most vulnerable to breakdown, fraud 
and abuse, egregious diseconomies, and the distortion of intended pur- 
pose. It is also helpful to know about those parameters that matter most 
when we try to upgrade a production system from mere adequacy to 


ce 


performance levels we might think of as “excellent” (see Part III, “Smart 
(Best) Practices’ Research”). 

Another way to look at production models is through optimization 
lenses. Operations research models—such as queuing, inventory man- 


agement, Markov processes—are relevant here.” 


5. For a good, brief discussion, see Stokey and Zeckhauser (1978) and Victorio (1995); 
also see the models, particularly that of case management, in Rosenthal (1982). 
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Evolutionary models. An evolutionary model describes a common pro- 
cess of change over time. It is constructed of three important subpro- 
cesses: variation among competitors, selection, and retention. Suppose, 
for instance, that in an agency enforcing health-related standards in the 
workplace, the complaints disproportionately concerned visible and 
annoying problems that were not, however, as hazardous to worker 
health as less visible and annoying problems. In this case, the evolu- 
tionary model suggests several plausible intervention points. The 
agency might try to educate workers to detect and complain about 
more serious problems, contriving thereby to swamp the less serious 
problems—thus changing the pool of “competitors.” It might start 
screening the complaints for their likelihood of being associated with 
more fruitful targets—thus changing the “selection mechanism.” Or it 
might attempt to persuade workers, and perhaps their union represen- 
tatives, to reduce their propensity to complain about matters the 
agency wishes to hear less about—thus changing the “retention mecha- 
nism,” workers’ attitudes.® 


Conceptualize and Simplify the List of Alternatives 


The final list of alternatives—the one you include in your presentation to 
your client and other audiences—will almost certainly look quite differ- 
ent from the one you started with. Not only will you have thrown out 
some that just don’t look very good, but you will also have done some 
work to conceptualize and simplify alternatives. 


mantic Tip Bye key to conceptualization is to try to sum up the basic 
strategic thrust of an alternative in a simple sentence or even a phrase. This 
is difficult but usually worth the effort. It helps to use very plain, short 
phrases stripped of jargon. When the Environmental Protection Agency 
(EPA) was created, the first administrator confronted (a partial list of) 
alternatives that might have been described as thus: “Let the states do the 
work; let the feds give them the money”; “Remove impediments to firms 
cooperating on antipollution research”; and “Sue the bastards” (meaning 


6. For other ideas and an excellent discussion of the uses of models generally, see 
Lave and March (1975). 


22 A PRACTICAL GUIDE FOR POLICY ANALYSIS 


the large, visibly polluting firms and industries, the prosecution of which 
would help build political support for the new agency). 

The key to simplification is to distinguish between a basic alternative 
and its variants. The basic element in many policy alternatives is an inter- 
vention strategy—such as regulatory enforcement or a subsidy or a tax 
incentive—that causes people or institutions to change their conduct in 
some way.’ But no intervention strategy can stand alone; it must be imple- 
mented by some agency or constellation of agencies (perhaps including 
nonprofit organizations), and it must have a source of financing. Usually 
the variants on the basic strategy are defined by different methods of 
implementation and different methods of financing. 

The distinction between a basic strategy and variants based on imple- 
mentation details is especially helpful when you have a lot of possible 
solutions to consider and you need to reduce the complexity involved in 
comparing them. Making the distinction puts you in a position to break 
your analysis into successive steps. In the first step, you might compare, 
say, three basic alternatives while ignoring the details described by their 
variants. Then, once you have decided on one of these basic alternatives, 
you could turn to comparing the variants. 

For example: You want to decrease the prevalence of heroin use in 
your county by 50 percent over the next five years.* You consider three 
basic alternatives: methadone maintenance, law enforcement pressure, 
and drug education. Potential variants for each one have to do with the 
funding sources, in that state, federal, and county money can be used in 
different degrees (although not all mixes of funds available for one 
approach are also available for the other two). Variation is also possible 
according to who administers the program(s): nonprofit organizations, 
county employees, or state employees. Or, you might consider variants of 
scale and scope, such as two possible sizes for your methadone program. 


7. Often, though not always, the basic element is something like a smart practice— 
that is, an intervention strategy that attempts to take advantage of some qualitative 
opportunity to create valued change at relatively low cost or risk. See Part III, “‘Smart 
(Best) Practices’ Research.” 

8. Choosing a numerical target can help to focus energies and can force you to think 
about what effects are too small to be worth seeking. But when all increments are of 
equal value, choosing a target may be arbitrary and self-defeating. 
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Points on a Continuum as “Alternatives” 


Suppose you are asked to recommend changes in, say, the rental rates for 
public housing in your city. Theoretically, each penny change in the rent 
charged could represent an alternative, but clearly that is a mistaken way 
to consider “alternatives.” A better approach is to make this into a two- 
step problem. Step one is to establish the upper and the lower limits of 
an acceptable range of possibilities, and step two is to choose some point 
within that range. Choosing each of these limits is a small policy problem 
in itself, complete with criteria, projections, and the like. For instance, 
equity might require that the upper limit not be “too high,” meaning 
somewhere close to $600 per month, whereas affordability might suggest 
a slightly lower upper limit. Cost recovery requirements might suggest a 
lower limit of, say, $450 per month. In any case, suppose that at the end 
of step one, the acceptable range has been narrowed to $475-$575 per 
month. One might almost say that a good move for step two is simply to 
take the midpoint of these two limits, $525 per month. But there might 
be additional criteria of interest, for example, finding a “reasonable” 
increment relative to the current rental rate. If the current rate is $475 per 
month, a $50 increment, to $525, could be seen as reasonable, but so 
might a $75 increment (especially if rents have not been raised in several 
years), which will permit the city’s housing authority to offer some 
needed services to residents. At any rate, $25 increments between $450 
and $575 seems to be the psychologically “right” set of alternatives—not 
too large and not too small for the range of options to be considered. 
Thus, in the end, we have narrowed our alternatives down to six, from an 
initial array of several thousand. 

This two-step procedure could be useful for a variety of problems 
involving near-continuous variables as alternatives, for example, budget 
allocations, future dates to begin or to discontinue a service, the number 
of people to be accommodated by some project or program, emission 
limits for some effluent, fee or fine schedules, or quantity of water to be 
released from a reservoir. 

The great majority of social science hypotheses about what might 
work to ameliorate a given problem show up in the language of continu- 
ous variables, which then need to be transformed by the policy analyst 
into policy-compatible discontinuous choices. If, for instance, studies 
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show that the price elasticity of a pack of cigarettes is —.4, that tells you 
about a continuous relationship (within a certain range) between aggre- 
gate cigarettes demanded and the price charged. But if you want to 
exploit this fact to raise cigarette taxes so as to discourage smoking, you 
need to translate this information into particular numbers, for example, 
“Raise the tax $.25 per pack to $1.75.” 


Design Policy Alternatives 


This handbook assumes throughout that you are working on a problem 
of policy choice. However, a special case of policy choice occurs when 
you wish to, or have to, design at least one policy alternative to add to the 
menu of possibilities. Perhaps you are just not satisfied with the menu of 
alternatives that people in the policy environment are already talking 
about. Most challenges in “exploiting opportunities,” as opposed to “solv- 
ing problems,’ also involve designing some relatively fresh or new system 
for doing so. 

What is the essence of a design problem? It starts by specifying a pri- 
mary objective to be achieved, and by assuming that many different ele- 
ments need to be combined into some workable system that would 
achieve it. Think, for instance, of an objective like “increase patronage in 
our regional mass transit system by 20 percent over the next three years.” 
You might approach this design problem in the following way: 


1. The system being designed to meet this objective involves elements 
like (1) quantity of buses and subways, (2) their schedule, (3) tolls 
on local highways, (4) rules about diamond-lane usage, (5) parking 
meter rates, (6) monthly bus pass charges, (7) arrangements between 
transit operators and large traffic-generating institutions for pass 
sales to employees, and (8) no doubt, a host of other elements. 

2. Those elements could be combined in a (sometimes large) variety 
of ways. To complicate matters even further, each element is also in 
need of design analysis. 

3. As hypothetical combinations shift in the mind of the designer—or, 
more likely, in the minds of the group of stakeholders involved in 
the design process—the hypothetical system could also shift the 
relative significance of the varied objectives, including the primary 
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one that motivated the design effort in the first place. The 20 per- 
cent goal might appear to be too ambitious, given the budget con- 
straints on bus purchases or union resistance to bus driver schedule 
changes. By contrast, newly salient environmental protection goals 
might emerge to strengthen commitment to the 20 percent goal. 

4. The process of combining and recombining elements into different 
versions of a workable system is based largely on trial and error. For 
example, the thought process involved might proceed in this man- 
ner: “Okay, let’s start by seeing how far we can get by discounting 
monthly bus passes. If that’s insufficient, let’s see how much more 
we can get by adding time-of-day pricing. And what if we raise the 
tolls on some of the key highways and choke points? Maybe that 
will be better than tinkering with the pricing of bus service... .” 


Is there a way to minimize the thrashing about that often accompanies 
this trial-and-error process? And how, if at all, might the Eightfold Path 
framework contribute? 


Consider targets and budgets. You can eliminate a lot of thrashing by 
artificially limiting either the budget you can work with or the target you 
wish to achieve. In the preceding example, we discussed a 20 percent 
increase-in-ridership goal and worked backward from there to design the 
least-cost (in a broad sense) way of getting there. This time, let us take the 
opposite tack of assuming a particular budget, as in, “If your agency has 
$10 million to spend on emergency preparedness next year, what would 
be the most effective way of spending it?” The budget of $10 million is 
chosen somewhat arbitrarily, although it is expected to be somewhere 
within the realistic range of possibilities. But it has the advantage of focus- 
ing your imagination for designing a spending package on something 
concrete. If at the end of the analysis, you believe $15 million might be a 
more realistic and politically achievable number, then think about what 
an extra spending package of $5 million would look like. In any case, it is 
not realism so much as analytic focus that you are trying to achieve. 


Look around. Take advantage of design efforts made by others before 
you. Perhaps the problem you are dealing with is so new or unique that 
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you will be the first, or even the only, person to oversee the needed design 
work. More likely, though, others have already dealt with this problem. It 
pays to see what they have done and to assess their degree of success or 
failure. Successful approaches are usually the most helpful, but some- 
times you can learn a lot from evident failures, as well. 

Where to look? It may help to observe sister jurisdictions or institutions. 
If you are thinking about a problem at the state level, look to other states; 
at the city level, to other cities; at the community foundation level, to other 
community foundations. Professional associations linking government 
officials (such as chief state school officers, district attorneys, or county 
welfare directors) often publish materials describing “best practice” in one 
or more of their member jurisdictions; even if they do not, a phone call to 
the executive offices of the association may produce useful leads. 

When you are looking around, however, you may need to consider 
whether the problems in your “target” jurisdiction and the “source” juris- 
diction are similar in nature and scale. A city that has nearly solved its 
homeless problem with a service-rich mix of supportive housing and 
solicitous outreach (e.g., Philadelphia) may or may not be a source of 
good ideas for a city with a problem that is four or five times as large per 
capita and a physical climate that is very mild and therefore attractive to 
homeless people (e.g., San Francisco). You may discover that although the 
source’s ideas are very good indeed, they will need to be adapted to the 
target jurisdiction’s particular context. (For more on how to deal with this 
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“extrapolation problem,” see Part III, “‘Smart (Best) Practices’ Research.”) 


Start wisely. Your initial step is important, for the design process is some- 
what “path dependent.” That is, you may see, or not see, relationships and 
possibilities differently depending on when and how in the trial-and- 
error process you come upon them. For better or worse, there are several 
different ways to choose a first step. Hunch and experience are probably 
more important in making this choice than any strictly logical or ana- 
lytic guidance of which I am aware. Consider the following possibilities: 


e Start with the least flexible design element. In our mass transit exam- 
ple, you might assume that budget constraints on the purchase of 
new rolling stock foreclose a large number of design options. 


THE EIGHTFOLD PATH 27 


e Start with the most “powerful” design element. By powerful, I mean 
that the element, once chosen and its likely value assumed, strongly 
suggests the next most important element to consider. Once those 
two elements are chosen, the rest more or less “naturally” fall into 
place. To do this, you need a realistic theory of cause-and-effect 
that, to some extent, links the various elements. Although a perfect 
theory is probably lacking, some rough guesses can usually suffice. 
In the mass transit case, I would guess that supplying disincentives 
to single-occupancy auto use, in the form of tolls or congestion fees, 
would probably be the best place to begin, the theory being that 
mass transit use depends on making private autos a comparatively 
less desirable mode of transportation. That theory could also sug- 
gest making mass transit more attractive, perhaps by decreasing 
fares. But my own hunch is to start with disincentives to auto use, 
which I suspect would be more powerful than making mass transit 
marginally more attractive. Of course, on inspection, the opportu- 
nities for increasing the costs of auto use in targeted routes might 
be too few to make much difference. 

e Start with the most robust element. Since there is always a possibility 
that the complete, and ideal, system will never come into being— 
politics and budgets being volatile and uncertain—it might be a 
good idea to put something in place that would be socially valuable 
all by itself. This might, for instance, be some sort of limited set of 
elements (a subsystem, one might say) involving discounted bus 
passes for employees of large traffic generators. 

e Start with the most transitory, and least costly, element. I am thinking 
here of grant-in-aid windfalls that typically come from a higher 
level of government and that might, by design or by accident, disap- 
pear in a year or two. 


Check your assumptions as you proceed. The trial-and-error process 
requires, first, creative imagination and, second, rigorous evaluation of 
what creative imagination comes up with. The most commonly applied 
evaluative procedure is to generate what are known as “logic models.” This 
involves spelling out in some detail, and often with the help of graphic 
aids, how the emerging system is supposed to work. Presently, a leading 
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source of ideas about how to do this is the W.W. Kellogg Foundation 
“Logic Model Development Guide,” (available online at www.wkkf.org). 
The Kellogg Foundation’s approach is to divide the planned program or, 
in my language, system into six parts: resources/inputs, activities, outputs, 
outcomes, and impact. The modeler is supposed to fill in the details in 
each category and to specify the assumed relationships. In general, this 
approach assumes a production model, in which services of some kind are 
being delivered to a recipient population by a governmental or nonprofit 
organization. The model can be applied to nonproduction systems too, 
for example, regulatory systems, provided one is creative enough with fill- 
ing in the “activities” section, which works as a catch-all for the complex- 
ity of relationships among system elements. 

Another, very similar, approach, designed explicitly for service sys- 

tems, is “service blueprinting.” It goes somewhat beyond logic models in 
that it also asks the modeler to specify assumptions about “offstage” and 
back-office support functions.? 
Design problems usually have two stages. These stages are (1) design the 
system as it is projected to run in its steady state, and (2) plan the strategy 
of change that would take us from here to there. The first stage is predomi- 
nantly technical; the second, predominantly political and bureaucratic. 


Designing a case processing system. Design problems are generally of 
two types. One involves the management of “cases,” by which I mean 
individuals or other entities (such as firms or communities or lower lev- 
els of government) that receive some kind of “treatment.” The treatment 
may involve delivery of a subsidy, regulatory imposition of obligations, 
or application of some sort of person-changing regime (such as educat- 
ing children or getting offenders to “go straight”). The second principal 
type of design problem involves operating on a collectivity of some kind 
rather than on individual cases—for example, improving traffic flow, 
eliminating corruption in the police department, preserving habitat, or 
launching a community clean-up campaign. 


9. A good source of information about this approach is Bitner, Ostrom, and Morgan 
(2008). 
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The second type is too varied to discuss here, but a program that man- 
ages cases fits a rough template. That is, we can lay out a general proce- 
dure and list questions that should be asked. 

I use the term program deliberately, to refer to an organized ensemble of 
routines. For instance, a program to distribute subsidies has routines for 
determining eligibility, calculating the amount to be paid, and detecting 
and deterring fraud and abuse. A regulatory program has routines for 
enforcing compliance with its rules, including inspection procedures and 
formulas for applying sanctions. It may also have routines for adopting 
rules, giving technical assistance to regulated parties, and offering forbear- 
ance in exchange for more efforts to cooperate. In a person-changing pro- 
gram, the routines typically bring the subjects into a setting where change 
is to be rewarded, facilitated, induced, or demanded, and where profession- 
als apply a whole kit of tools to the change process. Think of schoolchil- 
dren, classrooms, and teachers; or of patients, hospitals, and doctors; or of 
welfare recipients, training programs, and caseworkers and trainers. 

The logic model and service blueprint approaches can work for these 
types of programs, but each approach needs to be amplified to take 
account of two design levels, the individual and the population. 
Presumably, the analysis has to be done for each of these two levels sepa- 
rately. For instance, in a weatherization program, design issues at the 
individual case level might involve rules covering eligibility, what sort of 
weatherization measures to use in a particular type of dwelling unit, 
copayments (if any) that the agency charges the customer, and what sort 
of guarantees (if any) to provide the customer regarding performance. At 
the population level, another set of design issues intrudes, such as which 
type of customers to target, how to allocate weatherization-counselor 
time among target groups, whether to manage counselors by geographi- 
cal districts or by types of functional expertise, how to handle customer 
complaints, and the like. 

In confronting the inevitable design trade-offs at the individual level, 
it helps to look at any set of routines from two perspectives: that of the 
case manager in the agency and that of the citizen whose case is being 
“treated.” It often happens that routines designed to make life easier for 
program staff only make life harder for citizens. (“Sorry, we don’t give 
advice about that; send in the application and we'll respond ... .”) 
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It also helps to remember occasionally to go back to basics, to reiterate 
to yourself and others the main objective of the program. What social 
problem is supposed to be ameliorated? Or what existing program is to 
be redesigned to accomplish what objective better? Doing so presents an 
opportunity to think also about an often-neglected but very important 
design issue of a more instrumental kind: what evidence will you system- 
atically collect in the course of normal program operations that can let 
program managers know whether they are succeeding? That is, what 
tracking and evaluation routines can be designed and put in place? 

Another common set of design issues revolves around making 
adjustments—sometimes large, sometimes small—in an existing organi- 
zation or interorganizational network, so as to improve performance. 
Space precludes discussion here of this vast set of topics, but Appendix C, 
“Understanding Public and Nonprofit Institutions,” provides a menu of 
questions that any analyst of organizational performance issues ought in 
most cases to consider. 


A little help from your friends—and enemies, too. In some cases, the 
policy analyst works on the design problem more or less alone, like some 
brooding master architect. More likely, she does her work in loose or 
tight conjunction with other policy professionals who bring to the table 
different sorts of expertise (e.g., legal, engineering, fiscal) and who bring 
different viewpoints and priorities, as well. In any case, sooner or later, 
the design work will be held out for much more public view. Interested 
stakeholders, and perhaps more diverse audiences, who have previously 
been unaware of the design work going on seemingly behind the scenes, 
will see what youre up to. And they will offer their reactions. 

You will want to use such reactions for two purposes: to improve your 
design according to criteria that you and your client—and very likely your 
audiences—think are important, including the criterion of political feasi- 
bility; and to respond in such a way as to increase the political support (and 
decrease the opposition) that may come your way, now or later, on process 
grounds alone. I do not discuss here the strategy and tactics involved in 
communicating effectively with different audiences or the sequence in 
which to do so. I limit discussion to the questions of just how rough or 
polished the design should be that you first subject to relatively public 
review and comment and how tentatively you should put it forward. 
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Not surprisingly, a middle ground is best. A very rough and admittedly 
tentative design may leave out important points, creating a sort of vacu- 
um that outside interests will rush to fill on their own terms. You will then 
be forced onto the defensive, as you try to forestall the solution they have 
been first to suggest. Moreover, a very rough design may signal that the 
design work is at such a preliminary stage that it is not worth the trouble 
(or the risk of early-mover vulnerability) for any of the stakeholders to 
react at all. On the other hand, an overly polished and seemingly definitive 
design may signal to stakeholders that you are not interested in consulting 
them. In that case, they may feel that they have no choice but to oppose 
your design more vehemently than they otherwise might have done— 
unless, of course, they conclude that they have no choice but to get on 
board and negotiate for the best terms they can manage. 

Assuming that you have put out a rough-but-not-too-rough design 
and elicited a range of fairly thoughtful opinions as a result, you will 
need ways to keep in touch with the variety of actors who now expect— 
and whom you may wish—to be part of an ongoing, if rather diffuse, 
design process. Keeping in touch will require a communications infra- 
structure (telephone, fax machine, e-mail, chat room), of course. It will 
also require efforts on your part to develop the sort of network rela- 
tionships that permit rapid and reasonably trustworthy interpersonal 
communications. 

At a more analytic level—because any design must be anchored in 
working assumptions about its objectives, available resources, and 
constraints—you should choose your assumptions with an eye to their 
reasonableness as “a basis for further discussion.” You may feel some 
discomfort at putting forward such assumptions because they are hypo- 
thetical or speculative, and because critics might therefore challenge 
them as “lacking in rigor.” Policy analysis is not just an exercise in truth- 
telling, however. It is a pragmatic and responsible effort to facilitate 
reasonable discourse about a policy future that is inherently uncertain. 


STEP FOUR: SELECT THE CRITERIA 


It helps to think of any policy story (see Step Eight) as having two inter- 
connected but separable plotlines, the analytic and the evaluative. The 
first is all about facts and disinterested projections of consequences, 
whereas the second is all about value judgments. Ideally, all analytically 
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sophisticated and open-minded persons can agree, more or less, on the 
rights and wrongs in the analytic plotline and on the nature of its 
residual uncertainties. But this is not true with regard to the evaluative 
plotline—where we expect subjectivity and social philosophy to have 
freer play. The analytic plotline will reason about whether X, Y, or Z is 
likely to happen, but it is in the evaluative plotline that we learn whether 
we think X or Y or Z is good or bad for the world. 

This fourth step in the Eightfold Path belongs primarily, though not 
exclusively, to the evaluative plotline. It is the most important step for 
introducing values and philosophy into the policy analysis, because 
some possible “criteria” are evaluative standards used to judge the good- 
ness of the projected policy outcomes that are associated with each of 
the alternatives. 

Of course, the most important evaluative criterion is whether or not 
the projected outcome will solve the policy problem to an acceptable 
degree. But this is only the beginning. After all, any course of action is 
likely to affect the world in many ways, some desired and some not. Each 
of those effects—or projected outcomes, to apply our Eightfold Path 
language—requires a judgment on our part as to whether or not and why 
it is thought desirable. Our set of criteria embodies such judgments. 
Because any significant impact cries out for such a judgment to be made, 
the greater the variety of significant impacts, the richer will be the set of 
evaluative criteria needed to deal with them. 


AAN] Evaluative criteria are not used to judge the alternatives, or 
at least not directly. They are to be applied to the projected outcomes. It 
is easy to get confused about this point—and to get the analysis very 
tangled as a result. This confusion is encouraged by a commonsense way 
of speaking: “Alternative A looks to be the best; therefore, let’s proceed 
with it.” But this phrasing ignores a very important step. The complete 
formulation is “Alternative A will very probably lead to Outcome O,, 
which we judge to be the best of the possible outcomes; therefore, we 
judge Alternative A to be the best.” Applying criteria to the evaluation of 
outcomes and not of alternatives makes it possible to remember that we 
might like O, a great deal even if, because we lack sufficient confidence 
that A will actually lead to O,, we decide not to choose Alternative A after 
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all. With that judgment on the table, it will be possible to look for other 
alternatives with a greater likelihood of producing Oy. 


Commonly Used Evaluative Criteria 


Efficiency. Typically, the efficiency criterion is the most important evalu- 
ative consideration in cost-effectiveness and benefit-cost studies. I use 
efficiency more or less as the term is used in economics, for maximizing 
the aggregate of individuals’ welfare as that welfare would be construed 
by the individuals themselves—in economic jargon, “Maximize the sum 
of individual utilities, or “Maximize net benefits.” Another roughly 
equivalent formulation would be “Maximize the public interest.” 

Although efficiency has an antiseptic, technocratic, and elitist ring to it, 
the insistence here that “utilities” are to be assessed according to individ- 
ual citizens’ construction of their own welfare is thoroughly democratic. 
Indeed, siding with efficiency—on average, across most policy issues and 
policy decisions—is a way to produce more humanistic policy results, too. 
The reason is not that efficiency is so very humane a concept in itself, but 
that policy decisions failing to consider efficiency very often fail to take 
account of the welfare of the little guy at all. The little guy may be little, 
but in a proper efficiency analysis, he at least shows up to be counted. 
Efficiency analysis imposes a moral check (for whatever that is worth in 
the real world of politics) on political visionaries eager to relocate entire 
populations so as to make room for dams, and on special interests eager 
to impose seemingly small price increases on large numbers of consumers 
through protectionist measures in order to maintain the incomes of a 
relatively small number of producers. 

We should observe, though, that from the point of view of social jus- 
tice, the efficiency criterion may be somewhat limited. First, because 
analysts typically estimate people’s “utility” by inferring their willingness 
to pay for some benefit (or to be spared some deprivation), individuals 
with less money do not, in an analytic sense, have as much clout as those 
with more. Just how big a limitation this analytic anti-egalitarianism 
turns out to be will depend on particular cases, however. Second, if the 
values at stake have few or no human defenders, and therefore no human 
pocketbooks to back an estimate of willingness to pay, the efficiency cri- 
terion may underestimate these values even if by some conception of 
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justice they ought to be weighted heavily. In theory, ecological values are 
the main example, although in fact some ecological values do have 
human defenders who derive enormous utility from preserving them—a 
utility that would be accounted for in a proper efficiency analysis. 

Although cost-effectiveness analysis and benefit-cost analysis sound 
alike and are frequent traveling companions, they are not the same, and 
their uses can be quite different. True, both construe the policy problem 
as involving some production relationship between resources and 
objective(s). And both entail thinking about the relationship by using an 
economizing lens. However, cost-effectiveness analysis is usually satisfied 
to assess only the nature and quantity of the desired outputs, whereas 
benefit-cost analysis goes a further step and tries to estimate a value for 
those outputs in some fashion, typically in terms of money or (rarely) 
actual utility. Because it is less ambitious, the cost-effectiveness approach 
is more common in policy analysis than is the benefit-cost approach. 
Indeed, a surprisingly large number of policy issues can be simplified and 
stylized as cost-effectiveness problems, even though on the surface they 
may not appear to be likely candidates at all for this sort of treatment. 
Here are two examples: 


e The Mudville mayor wishes to respond to business complaints that 
building permits “take forever” to obtain. Given that you can spend no 
more than $500 and are permitted to change the work flow in the city 
planning office but not personnel assignments, the cost-effectiveness 
framework might suggest minimizing delay (measured in days) aris- 
ing from purely procedural and bureaucratic sources. 

e Quake City must upgrade the seismic safety of several thousand 
buildings constructed of unreinforced masonry. You have a twenty- 
year time span and no immediate budget constraint, but you wish 
to accomplish the job with minimum disruption to the lives (and 
incomes) of the residents and small businesses that may be dis- 
placed temporarily by the building renovation process. To mini- 
mize such disruption, cost-effectiveness analysis might lead you to 
propose that the work be done in one season rather than another, 
or that not all grocery stores be closed at once, or that tenants be 
assisted in organizing mutual-aid groups. 
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Relative to the benefit-cost approach, a cost-effectiveness framework 
typically simplifies policy analysis in another useful way, as well: it 
assumes as fixed either resources or outputs, and focuses only on choices 
involving the other member of this pair. Fixed resources usually involve a 
money budget or a human or physical asset such as a work team or a set 
of hospital beds. A fixed output is generally a target of some kind, such as 
a minimum required pollution abatement level or a maximum acceptable 
proportion of children failing an achievement test. Analysis then involves 
finding the best means to manipulate the other member of the cost- 
effectiveness pair so as to improve productive efficiency. Colloquially, if 
resources are fixed, you are “getting the biggest bang for the buck,” or if 
you have a fixed target, you may be “doing no worse with less.” 

Now suppose that, once you have figured out some approach where- 
by you can do no worse with less, you want to broaden your inquiry to 
explore whether you can make use of this new and better approach to 
produce a little more than you had originally planned. That is, instead 
of assuming that either resources or outputs are fixed, you are prepared 
to allow the scale of the activity to increase. The analytic challenge is 
much more difficult now, because at this point you cannot avoid the 
question of whether the augmented output “is worth it,” given the envi- 
sioned cost increment. That question cannot be answered unless you 
compare the utilities of both the cost increment and the augmented 
output. That is, cost-effectiveness analysis must now rise to the level of 
benefit-cost analysis. 

Here is an example of a 1995 RAND Corporation cost-effectiveness 
analysis—concerning crime prevention strategies aimed at juveniles— 
that tried, rather convincingly, to stretch its way into a benefit-cost analy- 
sis without quite admitting it. The study compared the cost-effectiveness 
of four “early intervention” strategies to that of incarceration under 
California’s “three-strikes” law and found that at least two of them were 
very likely superior. It then reasoned: 


It might be inferred from California’s vote in favor of the three- 
strikes law that the public believes a 21 percent reduction in 
crime [RAND’s estimate from an earlier study] is worth the 
measure’s cost of $5.5 billion a year. For less than an additional 
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billion dollars, graduation incentives and parent training could 
roughly double that crime reduction, if they are as effective as 
our analysis suggests.'° 


Equality, equity, fairness, justice. There are, of course, a great many dif- 


ferent, and often opposed, ideas about what these terms do, or should, 
mean. In addition to thinking hard about these ideas yourself, sometimes 


you should also take your audience through some of that thinking, as in 


the following examples: 


e Drivers who do not carry liability insurance leave persons whom 


they injure in auto accidents at risk of being undercompensated. 
Many of those who “go bare” are relatively poor. Many other drivers 
purchase their own insurance against exactly this risk (“uninsured- 
motorist coverage”). A policy proposal to pay for all drivers’ liabil- 
ity insurance out of a fund created by surcharges at the fuel pump 
was denounced by some observers as “inequitable” to the poor, who 
were going bare of insurance. Other observers said that those who 
go bare impose inequitable premium expenses or risks of under- 
compensation on the rest of society, including many individuals 
who are themselves poor or not very well off. Clearly, the analyst 
needs to include a discussion of the idea of equity. 

The current debate over whether to retain affirmative action prefer- 
ences for African Americans and certain other minorities in univer- 
sity admissions is sometimes said to pit fairness to individuals 
against justice to social groups. This is odd, though, since some 
philosophers—and most ordinary folk, too—suppose that no sys- 
tem claiming to be just could contain any features deemed unfair. 
Again, the analyst has a job to do in sorting out ideas and language. 


Freedom, community, and other ideas. To stimulate thought, here is a (far 
from complete) list of more ideas of possible relevance as evaluative crite- 


ria: free markets, economic freedom, capitalism, “freedom from govern- 


ment control,’ equality before the law, equality of opportunity, equality of 


10. See Greenwood et al. 1995. 
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result, free speech, religious freedom, privacy, safety (especially from 
chemicals, various environmental hazards, and the like), neighborliness, 
community, sense of belonging, order, security, absence of fear, traditional 
family structure, egalitarian family structure, empowerment of workers, 
maintenance of a viable nonprofit sector, voluntarism, and trust in others. 


Process values. American democracy values process and procedure— 
that is, having a say in policy issues that affect you, rationality, openness 
and accessibility, transparency, fairness, nonarbitrariness—as well as 
substance. These considerations probably apply to the very design or 
decision process for which you are doing your present analytic work. 
Therefore, remember to consult broadly and equitably. In addition to 
building up legitimacy for your work, you may be surprised at how much 
you can learn, especially from people who are very unlike yourself social- 
ly or ideologically. This does not, of course, mean that you should in the 
end accord equal deference to all opinions or desires, or keep the consul- 
tative process open forever. Some opinions are more creditable than oth- 
ers, and at some point consultation must give way to decision. 

Do not make the mistake of thinking that “more participation” or 
“greater access to the process” necessarily equates to “more democratic” 
or “more rational.” Greater opportunities for participation may be 
exploited more heavily by those with more time to participate or by those 
with special interests to protect or by ideological zealots. Ordinary people 
and their ordinary concerns can come out as relative losers. 


Weighting Conflicting Evaluative Criteria 


As we saw in the case of defining the problem, when values are at issue— 
as they are in regard to criterion selection, as well—we must reckon how 
to weight opposing values. There are three general approaches to this 
problem. 


The political process takes care of it. One approach is simply to allow exist- 
ing governmental and political processes to determine the weighting. 
Typically, this approach will accord primacy to the analyst’s employer or 
client, as well as allowing derivative influence to be exercised by those parties 
in the relevant arena who are in turn important to the employer or client. 
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The analyst imposes a solution. A second approach is for the analyst 
herself to modify—though not replace—the weighting assigned by the 
employer or client by reference to some overarching philosophical or 
political conception. The justification usually offered for this approach is 
that because certain interests, and perhaps philosophies, are typically 
“underrepresented” in government and politics, and because the analyst 
is in a better position than most other participants in the process to see 
or understand or appreciate this problem of underrepresentation, the 
analyst is duty-bound, or at least permitted, in the name of fairness and 
democracy, to right the balance. 

For instance, some observers would argue that were it not for policy 
analysts, efficiency-related criteria would rarely be heeded and that, as a 
consequence, analysts should in effect speak up for the taxpayers whose 
interests may be squeezed out by better-organized advocacy groups. A 
related argument is sometimes made that certain conceptions of equity— 
in particular those having to do with the idea that the beneficiaries of 
publicly provided goods or services should pay for them—are underrep- 
resented except among policy analysts. (These conceptions of equity 
typically exclude public expenditures deliberately intended to redistrib- 
ute wealth among citizens.) Other interests that people sometimes claim 
are underrepresented and therefore need representation by analysts are 
future generations, children, people who live outside the jurisdiction 
making the decisions, ethnic and racial minorities, women, the poor, 
consumers, and animals and plants (ecological entities). 

A variant of this approach introduces the idea of an educational pro- 
cess. Depending on circumstances, the analyst might encourage influen- 
tial political actors—perhaps including the analyst’s boss or principal 
client—to rethink their existing criteria in the light of facts or argu- 
ments the analyst can draw to their attention. In this case, the analyst 
takes responsibility for opening up a dialogue, and perhaps for trying to 
infuse it with reason and insight, but then allows the political process to 
take over. 


The distribution of “rights” precludes some solutions and forwards 
others. If X has a recognized property “right,” you can’t easily override 
it just because your policy solution would find that convenient; and if 
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Y has a “right” to privacy, you might be inclined to tilt the weighting of 
criteria heavily in that direction. Generally, claims based on rights are a 
reasonable guide to choosing “better” policies, and rights-based criteria 
deserve some extra weighting. However, plenty of exceptions exist, and 
it pays to examine, briefly, the whole matter of where rights come from 
and how policy analysis can make good use of them. This is a contro- 
versial matter, of course, and my thoughts on the matter are certainly 
contestable. 

Typically, rights are specially protected claims of an individual or a 
group against encroachment by “others,” including society as a whole, 
though in some cases it is society that claims the rights against compo- 
nent groups or individuals. Sometimes rights are long-standing, well 
established, consensual, and, within our social context, unquestioned, as 
in “X has a right to be treated with dignity, irrespective of X’s economic 
condition.” In these cases, the pattern of rights claims, hedged and lim- 
ited though they might be, very likely is found to be a good self-help tool 
for organizing the many and varied interactions in a complex society. But 
rights are sometimes more emergent than established, and claims based 
on rights can be quite contentious or in conflict: “I have a right to use my 
cell phone in any place, private or public” versus “I have a right not to be 
disturbed by your loud and obnoxious cell phone conversation, thank 
you very much.” 

It is best to think of all rights claims as emerging from a social process 
of trial-and-error and contestation, with the ones that seem obviously 
legitimate to us being merely the (so-far) best established and (probably) 
most socially beneficial. Claims that particular rights are justified by 
nature or “divine will” or reason or “our common humanity” are simply 
rhetoric, because these justifications are always challenged by others. 
Over the centuries through which these debates have continued, no per- 
manent resolution has occurred, and I think will never occur, since 
“rights” are simply convenient tools of social organization and rights- 
based claims, a consensually accepted way of negotiating the changing 
landscape of whose interests should be protected to what degree and with 
what exceptions. 

From the point of view of a hypothetical social engineer trying to 
improve social welfare, some rights should certainly be treated as relatively 
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fundamental. If the moral realm were the legal realm, these rights would 
be considered constitutional. But, like the Constitution, even fundamen- 
tal rights, such as “the right to privacy” or “the right to control your own 
body,” should evolve to fit new social and technological conditions. 
Technological change raises questions of privacy and transparency (e.g., 
confidentiality and fairness), and the past structure of rights is not neces- 
sarily a good guide to how to redesign that structure for the emergent 
situation. A fortiori, this applies to matters of lesser moment—where it 
is easier to see that rights are constructed rather than found. Forcing oth- 
ers to listen to your cell phone conversation may or may not be a right we 
wish to create and honor, but it is surely novel, and needs to be settled by 
reference not to reason and the like but rather to the balancing of utilities 
in a strictly pragmatic fashion. The same applies to compensation for 
takings, decent health care, privacy, abortion, and a host of other matters 
that now or in the recent past have been subject to debate over who ought 
to have what sort of highly protected positions that we dignify and crys- 
tallize as rights. 

Please do not misunderstand this as an argument in favor of relativ- 
ism, which in many people’s usage is the same as saying that there is no 
choosing among different rights claims, that one is as good as another. 
That is not true. Certainly, allowing people to claim a high level of 
protection for (i.e., a right to) certain values—such as individual 
autonomy—is beneficial to the running of a modern democratic soci- 
ety. But this right sometimes needs to have exceptions carved out of it 
to accommodate cases when the exercise of this right imposes exces- 
sively on other people. The fine texture of the fabric of such rights is 
always subject to discussion, and the basis for making these decisions is 
not to be found in rhetoric or in philosophical speculation but in the 
analysis of alternative fabrics, each taken as a whole and including all 
the internal tensions that are bound to be included in them. The evolu- 
tion of rights in the moral realm—that is, in the realm of private prac- 
tice and thence public opinion formation—involves the sort of con- 
stant tinkering and adjustment we see in the realm of both statutory 
and judge-made law. 

In the end, therefore, claims to weight criteria by reference to which 
rights ought or ought not to take priority deserve to be treated critically. 
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Semantic Tip| Note, however, that rights alleged to be “natural” or 
“human” are conceptually quite different from legal rights, despite the 
semantic similarity. Examples are the conflicting abortion stances predi- 
cated on right-to-life values or a woman’s right to control her own body. 
Alleged natural or human rights are sometimes controversial in that 
some people would like to have them recognized as legal prescriptions 
whereas others would oppose such recognition. 


Commonly Used Practical Criteria 


Not all criteria that come into play in an analysis are part of the evaluative 
plotline. Some are purely practical and are part of the analytic plotline. 
These criteria have to do with what happens to an alternative as it moves 
through the policy adoption and policy implementation processes.'! The 
main ones are legality, political acceptability, robustness under conditions 
of administrative implementation, and improvability. 


Legality. A feasible policy must not violate constitutional, statutory, or 
common law rights. Remember, however, that legal rights are constantly 
changing and are often ambiguous. It is sometimes worth taking a gam- 
ble on a policy that might—or might not—be adjudged illegal when 
tested in court. (In such cases, advice of counsel is clearly in order to help 
craft the policy so that its survival chances are enhanced.) 


Political acceptability. A feasible policy must be politically acceptable, or 
at least not unacceptable. Political unacceptability is a combination of two 
conditions: too much opposition (which may be wide or intense or both) 
and/or too little support (which may be insufficiently broad or insuffi- 
ciently intense or both). 

Do not take a static view of unacceptability, however. Always ask your- 
self the question “If my favorite policy solution doesn’t look acceptable 
under current conditions, what would it take to change those conditions?” 


11. I said earlier that criteria apply to outcomes and not to alternatives. However, this 
statement needs a slight amendment in the case of practical criteria, which apply not 
to outcomes but to the prospects an alternative faces as it goes through the policy 
adoption and implementation processes. 
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You may discover that creative political strategizing can open up options 
that haven’t been seriously considered before. (Discussion of techniques 
for building coalitions and launching successful campaigns is far beyond 
the scope of this book, but Appendix D, “Strategic Advice on the Dynamics 
of Political Support,” sketches some of the basics.) 

In assessing strategic limitations and possibilities, it will help to make 
use of various models of the political process. As I observed earlier, mod- 
els are based on metaphors, and the ones that are likely to be most valu- 
able, in this case are these: 


e A complex game in which well-organized and well-positioned 
minorities enjoy special advantages 

e A theater, in which the actors are elected officials who strive, with 
or without a basis in reality, to create a good appearance—to them- 
selves, to each other, to the critics, and to the audience (whose 
approval, ultimately, is all-important) 

e A marketplace of slogans, symbols, and ideas, with a mix of honor- 
able merchants and hucksters as sellers and a mix of sophisticates 
and innocents as buyers 

e A school in which elected officials learn how to do good policy 
design work and sometimes share their results and their methods 
with their classmates 


How exactly is one to make use of such models? Think of them as concep- 
tual lenses. Observe the relevant political process through each of them in 
turn, and identify the probable pitfalls and opportunities brought into 
focus by each.!? 


Robustness and improvability. Policy ideas that sound great in theory 
often fail under conditions of field implementation. The implementation 
process has a life of its own. It is acted out through large and inflexible 
administrative systems and is distorted by bureaucratic interests. Policies 
that emerge in practice can diverge, even substantially, from policies as 
designed and adopted. A policy alternative, therefore, should be robust 


12. An analogous procedure was first given prominence by Graham Allison (1971). 
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enough that even if the implementation process does not go very smoothly, 
the policy outcomes will still prove to be satisfactory.!4 

Some adverse implementation outcomes usually worth worrying 
about are long delays; capture of program or policy benefits by a rela- 
tively undeserving and unintended constituency; excessive budgetary or 
administrative costs; scandal from fraud, waste, and abuse that under- 
mines political support and embarrasses supporters; and administrative 
complexities that leave citizens (and program managers) uncertain as to 
what benefits are available or what regulations must be complied with. 

Even the best policy planners cannot get all the details right at the 
design stage. They should therefore allow room for policy implementers 
to improve on the original design. The most common vehicle for such 
improvement is participation in the implementation process by indi- 
viduals and groups whose expertise or point of view was not included in 
the design phase. However, the openness that makes for improvability 
can also, by opening the door to hostile political interests, diminish 
robustness. Hence, a careful evaluation of the current factual situation— 
personalities, institutional demands and incentives, political vulnerabili- 
ties, and so on—is usually in order. 

In estimating robustness and improvability, models of bureaucracy 
can serve as useful conceptual lenses, as suggested earlier with regard to 
carrying out political analysis. I find the most useful metaphors for 
bureaucracy to be these, listed in no particular order: 


e An automaton enacting preprogrammed routines (“standard oper- 
ating procedures,” or “SOPs”) 

e A person in an environment, driven by survival needs, self- 
enhancement interests, and, under some conditions, a desire for 
self-actualization 

e A political arena wherein individuals and factions jockey for influ- 
ence over the organization’s mission, access to its decision systems, 
and its prerequisites 


13. Robustness under conditions of “deep uncertainty” is sometimes a preeminently 
important criterion—for example, for long-term and very risky problems such as 
global climate change or shifts in the technical and organizational capacity of terror- 
ist movements and cells (Lempert, Popper, and Bankes 2003). 
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e A tribe with its own rituals and an array of safeguards against con- 
tamination by “outsiders” 

e Asociety of individuals cooperating toward a more-or-less common 
set of goals 

e A structure of roles and interrelationships that are intended to 
complement one another in a rational division of labor 

e An instrument used by “society” for society’s own objectives 


“Criteria” as Logical Constructs 


Criteria such as efficiency, equity, political acceptability, and robustness 
are substantive. But we can think of criteria of a purely formal sort, as 
well. For instance, we can distinguish among criterion values that we 
wish to maximize, those that must be minimally satisfied, and those of a 
generally lesser priority for which “more is better.” 

It is helpful to focus initially on one primary criterion, a principal 
objective to be maximized (or minimized). Typically, this principal 
objective will be the obverse of your problem definition. For instance, 
if your problem is that too many families are homeless, then your 
principal objective will probably be to minimize the number of home- 
less families. If the problem is that global temperatures are rising too 
rapidly, a good statement of a principal objective might be, “Minimize 
or reverse the increase of global temperatures.” Naturally, there are 
other criteria to judge outcomes by, such as costliness, political accept- 
ability, and economic justice, and these should all enter into the final 
evaluation. However, unless you focus—ainitially, at least—on a single 
primary criterion and array others around it, you will likely find your- 
self getting very confused. As you get deeper into the analysis and feel 
more comfortable with a multiplicity of important objectives, you 
may wish to drop your emphasis on a primary criterion and work on 
a more complex “objective function,” in the language of mathematical 
programming. 


Linear programming. A mathematical (and now computer-accessible) 
technique for optimizing choice when you have a principal objective or 
an objective function and a scarce stock of resources for maximizing it 
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is called “linear programming.” 14 Often, at least some of the resources— 
such as the agency budget and the available physical facilities promised 
by a nonprofit agency—are constrained. Even if the problem is not sub- 
ject to simple quantitative assessment, analysts often find it useful to 
take advantage of the logical structure of linear programming to con- 
ceptualize their task. The conventional formulation then sounds like 
this: maximize this objective (or objective function) subject to such- 
and-such resource constraints. 

Here is an example from the homelessness problem: “maximize the 
number of homeless individuals housed on any given night, subject to 
the constraints of not exceeding $50,000 per night total budgetary cost to 
Agency X, not putting shelters into Neighborhoods A and B for political 
reasons, and trying to give ‘more’ choice to the beneficiary population as 
to where they will take shelter.” 


semantic Tip] Linguistic clarity. If it is possible to sort your criteria 
according to whether they refer to values to be maximized, values that 
stand as constraints, or values that have a more-is-better quality, keep the 
different statuses of the criteria in mind. Be conscious of them. You can 
do this with a simple verbal trick: as appropriate, define your criteria as 
“maximize such-and-such value,” “satisfy such-and-such value constraint,” 
or “get more of such-and-such value.” If your criterion label contains no 
signal as to the better direction to move in, as in “governance structure” or 
“effect on landlords,” it is almost certainly insufficient. 

In any case, to the extent possible the criteria should be characterized 
both in conceptual and in operational (typically quantitative) terms. 
Conceptually, for instance, one talks about “maximizing the reduction of 
greenhouse gas emissions from publicly owned buildings,” whereas opera- 
tionally, one talks about “Minimizing the tons of greenhouse gas emissions 
per month from publicly owned buildings.” In this case, the operational 
definition is a close proxy for the more qualitative conceptual definition. 
Frequently, however, something of a gap exists, since what is measurable 
may only imperfectly reflect the conceptual characterization. For 


14. See Stokey and Zeckhauser (1978, chap. 11). 
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instance, minimizing “the hassle factor” to the citizen in recycling his 
garbage is conceptually meaningful but hard to express quantitatively. It 
is really about the psychology of effort, the degree of belief in the desir- 
ability of the goal, and the degree of frustration involved in preparing 
one’s garbage for pickup. In this case, the best you could probably man- 
age operationally would be to estimate the number of minutes the citi- 
zen spends per week to cooperate in the enterprise. 

To the extent possible, group your criteria in such a way that all the 
“positive” (benefit) criteria are clustered separately from the “negative” 
(typically, cost plus avoidance of negative unintended consequences) 
criteria. In a logical sense, how one does this does not really matter. But 
it makes for easier reading and discussion. It is a little like arranging your 
bridge hand by suit and, within suits, by number sequence. 

Don’t embrace euphemisms or other dodges as a substitute for words 
that describe harsh realities. The client for one student project asked for 
advice on what adult school programs to cut in order to save money in 
financially desperate circumstances. The students initially put together a 
brief defending adult school programming in general, at best leaving to 
inference what elements of the bundle were most deserving of cuts. 


Avoid Confusing Alternatives and Criteria 


SET 4 trernatives are courses of action, whereas criteria are men- 
tal standards for evaluating the results of action. How could you ever 
mistake an alternative for a criterion, or vice versa? As with many 
instances of confusion in policy analysis, the source of such a mistake is 
likely to be semantic. Consider, for example, a senior manager in a state 
regulatory agency dealing with worker safety. She wishes to incorporate 
worker complaints into the agency’s strategy for targeting inspections 
across work sites in the state. Her assistant presents her with a number of 
alternatives for doing so, one of which is called “Rapid-response (twenty- 
four-hour maximum) hotline.” Not surprisingly, one of the criteria for 
assessing outcomes is “responsiveness.” The alternative therefore seems a 
lot like the criterion. But this is an illusion. The alternative (course of 
action) is really the hotline. The main reason it looks like a criterion is 
that the intention of “rapid-response” has crept into the definition of the 
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alternative.! This is a dangerous mistake, because one should not 
assume through definition that an intention, as expressed in the verbal 
characterization of the alternative, will actually be realized. 

This sort of confusion is most likely to arise when the internal activi- 
ties of an organization are under discussion, since proposals to create or 
modify organizational units resonate with intentionality. Consider a pro- 
posal to create a performance measurement office, a strategic planning 
team, and a customer service department. The performance measure- 
ment office may end up, for whatever reasons, using meaningless mea- 
sures collected by unreliable agents; the strategic planning team may be 
deliberately ignored by savvy or possibly unsavvy managers; and the cus- 
tomer service department may unintentionally end up as an instrument 
of customer alienation. I once questioned a student’s proposal to create a 
“drug counseling service” for employees within an organization. The 
proposal seemed to me too weak to make a dent in the organization’s 
problem. The student countered, “No, I’m not talking about any old 
counseling service that might attack this problem, but an ‘effective’ one.” 
Nothing in the student’s account of how the service was to work increased 
the odds that it might really be effective. Effectiveness was assumed sim- 
ply because the student wished to assume it. 


STEP FIVE: PROJECT THE OUTCOMES 


For each of the alternatives on your current list, project all the outcomes 
(or impacts) that you or other interested parties might reasonably care 
about. This is the hardest step in the Eightfold Path. Even veteran policy 
analysts do not usually do it very well. Not surprisingly, analysts often 
duck it entirely, disguising their omission by a variety of subterfuges. 
Hence, the most important advice about this step is simple: do it. 

At least three great practical as well as psychological difficulties must 
be confronted here. First, “policy” is about the future, not about the past 
or the present, but we can never be certain about how the future will 
unfold, even if we engage it with the best of intentions and the most 
thoughtful of policy designs. 


15. Also, in this case, the stem respons- appears in both alternative and criterion. 
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Second, “project the outcomes” is another way of saying, “Be realistic.” 
Yet realism is often uncomfortable. Most people prefer optimism. Policy 
can affect people’s lives, fortunes, and sacred honor, for better or for 
worse. Making policy, therefore, imposes a moral burden heavier than 
many people care to acknowledge. Understandably, we would rather 
believe that our preferred or recommended policy alternative will accom- 
plish what we hope and that it will impose fewer costs than we might 
realistically fear. 

Third, there is what is sometimes called “the 51-49 principle.” That 
is, in the thick of the policy fray, we are driven out of pure self-defense 
to treat 51 percent confidence in our projection as though it deserved 
100 percent confidence, so that we sometimes mislead not only others 
but ourselves as well. The first difficulty—namely, that we can never have 
wholly convincing evidence about the future—compounds the second 
and third, inasmuch as our wishful thinking is not readily disciplined by 
reference to empirical demonstrations and proofs. 

These psychological difficulties notwithstanding, systematic efforts to 
project outcomes are essential. For policymakers in a modern democracy, 
neither following gut instincts nor reading pigeon entrails is a responsible 
alternative. 


Extend the Logic of Projection 


In this section I discuss, in a very general way, the logic of combining 
models and evidence to produce usable projections of policy outcomes 
from the various alternatives being considered. The logic is largely that of 
common sense but with some important additions. 

The first addition is that of metaphor. Policy analysis, as we have seen, 
makes use of the metaphors behind the models—metaphors such as 
“bureaucracy as automaton” and “politics as theater” and “this piece of 
the world as production system’—to yield qualitative insights about 
important causal relationships. The especially important relationships 
are those that may afford useful intervention points in complex systems 
or that present potential pitfalls in the policy adoption or implementa- 
tion process. 

Second, policy analysis uses social science to the degree that it can. A 
great deal of social science is directed toward answering the question “Is 
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Model X of this piece of the world realistic?” Social scientific studies of 
this type can often be useful for diagnosing the existence of problems, 
mapping trends, and deciding whether some seemingly “smart” practice 
(see Part III) is worth trying to replicate. You should be careful, however, 
to avoid using the social scientific standard of adequacy for judgments 
about the realism of a model, for it is quite conservative. In policy analy- 
sis the looser, but more appropriate, standard should be whether reliance 
on a model can lead to better results and avoid worse results than less 
disciplined guesswork. 

Third, policy analysis, as we have seen, uses multiple models. Most 
social science, in imitation of the hard sciences, looks for the “best” 
model (or, in the case of some practitioners, the “true” model). Because 
all models abstract from reality, however, even the best models are never 
complete. Although such abstraction may advance the progress of sci- 
ence, in the world of policy, where real consequences of policy choices are 
to be experienced by real people, no facet of a problem or the possible 
alternatives to be adopted can be exempted from analysis. Whatever 
models can be employed to illuminate some important facet of the prob- 
lem or of the possible outcomes should be employed—even if doing so 
results in an inelegant and ad hoc multiplication of subanalyses. 

Finally, even when you have adopted adequately realistic models of 
sufficient number and variety, these models still need to be used in con- 
junction with evidence about “initial conditions,” or the facts on the 
ground as they currently exist. For instance, “Deputy Director Smith is 
as incompetent as they come. The need to work around her will raise the 
risks of failure by at least 25 percent.” Or: “The community appears very 
angry about the drug scene right now, and they'll almost certainly help 
the police in the planned crackdown.” Although the projections of many 
models are not particularly sensitive to initial conditions, some are. 
These are the models that bear on projections of political acceptability 
and on the robustness of an alternative to the stresses of the implemen- 
tation process. 


Magnitude Estimates 


Projecting outcomes often requires you to think not just about the gen- 
eral direction of an outcome but about its magnitude, as well. Typically 
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it’s not enough to say, “We expect this program to have a very positive 
effect on reducing unwanted teenage pregnancies.” Instead, you'd want to 
say, “We expect this program to reduce by 100 to 300 the number of 
unwanted teenage pregnancies per year in this community over the next 
five years.” 

Sometimes a single point estimate of your best guess about the degree 
of magnitude will suffice. But in other cases, you should provide a range. 


Break-Even Estimates and the Problem of Uncertainty 


“You have no evidence this will work,” carp your critics. You—quite 
correctly—respond, “You have no evidence it wont.” You are both right, 
because “evidence” about events that have not yet occurred is a contra- 
diction in terms. Nevertheless, your critics make the valid point that you 
probably can’t be very certain that your recommended policy option will 
work and that the burden of justification (not, of course, a burden of lit- 
eral “proof”) falls on you.!® You will want to take up this burden using 
whatever strategic leverage you can muster. 

This means that you will set the bar of justification as low as is reason- 
able. Typically, you will want to claim only that the recommended course 
of action is “sufficiently likely” to produce results that are good enough 
to justify the known costs and risks. This approach is known as “break- 
even” or “threshold” analysis. It is an astonishingly powerful—yet simple, 
intuitive, and commonsensical—conceptual lens.!” It builds a decision 
framework out of what is known or reasonably assumed and handles the 
residual uncertainties by comparing them to elements in this more 
secure frame. 

Suppose, for instance, that some youth-guidance-oriented policy 
meant to reduce incarceration of juveniles is under consideration and 
has known costs of $1 million, but the level of effectiveness is specula- 
tive. You build a decision frame out of (1) a decision-rule that says, “If 
the benefits exceed the costs, do it,’ and (2) a known fact about the 


16. This assumes that you do make a recommendation. But even if you only lay out 
options and attach projected outcomes to them, you still cannot escape justifying the 
projections. 

17. It is also, I believe, a lens that is inadequately appreciated and utilized. 
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costs, $1 million. You then evaluate the remaining uncertainty in these 
four steps: 


1. Locate the point of minimum acceptable effectiveness given the 
costs. Ask: “What is the minimum level of effectiveness this policy 
would have to achieve in order to justify our spending $1 million?” 
Your answer: “Different observers have different opinions about 
how much avoiding an incarceration is worth, but leaving that 
aside and going with my own values, Pd say that a 15 percent 
reduction is the minimum I would accept given the expenditure of 
$1 million.”!8 

2. Referring back to your model of the processes that create the 
problem and hold it in place, ask: “What new processes, or chang- 
es in old ones, could conceivably produce this level of effective- 
ness?” This is largely a qualitative analysis. The answer might be: 
“Based on previous documentation of how the guidance process 
works, we can safely say that it works in different ways with differ- 
ent sorts of kids—when it works at all, that is. It can provide 
about half the kids more constructive life choices; in about a 
quarter of the cases it works through heightening the (realistic) 
perception of punishment; and in about a quarter of cases we are 
just crossing our fingers.” 

3. Assess how likely (or unlikely) it is that the processes for improve- 
ment thus identified will actually produce the required—that is, the 
break-even—level of effectiveness. It is particularly helpful to ask 
whether the break-even level (15 percent, in this case) looks like a 
plausible number given what is known or assumed about the effec- 
tiveness in similar circumstances of similar sorts of interventions. If 
the number is implausibly high, you might then go on to ask 
whether special circumstances of some sort might be at work in this 
case to help achieve it. Note that in this and the previous step you 


18. Some writers speak of “switchpoint analysis” and would refer to the 15 percent 
here as the switchpoint at which a decision maker would switch from a favorable 
view of this policy to an unfavorable view or vice versa. Others refer to “threshold 
analysis” and would call the 15 percent figure the threshold level of effectiveness we 
would need to assume in order to justify choosing this alternative. 
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must rely on what we might think of as “theory,” or self-conscious 
and evidence-based reasoning about the way causal processes work. 
Typically, these are the weakest links in the chain of policy-analytic 
reasoning. That is why it is particularly important—and particu- 
larly difficult—to take this step as thoughtfully, self-critically, and 
responsibly as possible. 

4. Estimate the probability of failure and the political and other costs 
of having to accept failure—asking yourself whether these costs 
would be tolerable should they be incurred. 


In the hope that it may be helpful to encourage readers to use 
break-even analysis (when appropriate, of course), I offer two more 
examples: 


e Policy X for establishing a chain of wildlife refuges looks like an 
excellent choice to implement a broader conservation agenda, pro- 
vided that the funding comes through as planned. But it might not, 
because federal grant-in-aid resources may not be forthcoming, or 
the governor may give the policy lower priority than she now 
promises, or some development interests that have their eye on two 
of the designated sites may find a way to block it. You interview 
your client, a state environmental agency director, and determine 
that she likes the program so much that she is willing to go for it if 
it has at least a 50-50 chance of working out. Your analysis can then 
focus her attention on why, after considerable research, you have 
concluded that it has a somewhat better (or somewhat worse) 
chance than 50-50, even though you may find it impossible to 
specify exactly how much better (or worse). 

e Building a new stadium for the Hometown Heroes looks like a good 
idea, given the nature of the costs and benefits, if average daily 
attendance turns out to be no less than 10,000. That’s the break- 
even attendance figure for you and the relevant decision makers. It’s 
up to them to decide, first, how confident they are that this break- 
even level will be reached and, then, whether that degree of confi- 
dence is enough to warrant making an affirmative decision. You can 
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thus organize your presentation of facts and opinions to focus on 
19 


these two key issues. 
semantic Tip| Assuming for the moment that benefits are uncertain while 
costs are not, ask yourself these two questions: (1) “Given what I know 
for sure about the costs of this alternative, what is the minimum help we 
need to get from Condition X to ensure adequately offsetting benefits?” 
and (2) “How reasonable is it to believe that Condition X will actually 
produce that minimum?” 


Try Sensitivity Analysis 


Which uncertainties are the most important, in the sense that relatively 
small changes in what you believe would cause you to change your mind 
about how desirable some alternative might be? By a process known as 
sensitivity analysis, you can discover these most important uncertainties. 
The procedures are somewhat technical (Morgan and Henrion 1990, 
chap. 8), but the intuition behind them is simple. Consider the several 
assumptions you have made on the way to your conclusion and suppose 
that each of them is somewhat mistaken. Now ask yourself: “How big a 
mistake can I afford in this assumption before this analysis is in really big 
trouble?” The smaller the affordable mistake, the more sensitive is your 
analysis to the particular assumption. (For a good example of sensitivity 
analysis, see Appendix A.) 

It is not hard to examine these assumptions one at a time. But what if 
they pile up in such a way that you are “somewhat” wrong on two or 
three or four assumptions all at once? This situation is typically dealt 
with by a technique called “Monte Carlo simulation,” which begins by 
recognizing that each assumption is in itself probabilistic and then com- 
bines the probabilities behind the assumptions to create a new set of 


19. A special case of break-even estimation is a fortiori estimation. If you hypothesize 
worst-case estimates of all important parameters that remain uncertain, and the 
policy alternative still satisfies your decision criterion, the alternative would, a for- 
tiori, prove satisfactory even if more careful estimates were to be more favorable. In 
that case, the more careful estimates are unnecessary. See MacRae and Whittington 
(1997) on a fortiori analysis (pp. 218-219) and, more generally, on the question of 
precision versus approximation in projecting outcomes (pp. 209-224). 
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probabilities about how the combination of assumptions will turn out. 
You can then say something like this: “Given the many possible scenarios 
that might occur, there is an 82 percent chance that the actual scenario 
would exceed our break-even requirement.””° 

But suppose that projections must be made for a future beset by mul- 
tiple uncertainties, like climate change or the global configuration of 
military forces and technologies twenty-five years off, for which proba- 
bility distributions are not known or are controversial. One promising 
approach makes use of any of an emerging set of computer-assisted pro- 
jection techniques, generally known as long-term policy analysis. This 
approach is similar to Monte Carlo simulation in that it starts with sce- 
narios about alternative futures, but instead it searches for policy choices 
that would be “robust,” in the sense that they would not necessarily be the 
best but would satisfy the whole, or nearly the whole, array of minimum 
policy desiderata. The objective is to minimize the maximum “regret” 


that relevant parties might experience.?! 


Confront the Optimism Problem 


Great ventures require optimism. Because even small ventures by govern- 
ment can affect so many lives, they are in their own way great. Hence, 
some realistic optimism is beneficial. But how do you guard against 
excessive optimism? 


Scenario writing. What scenarios might cause the proposal to fail to 
produce the desired outcome—that is, solving or sufficiently mitigating 
the policy problems? Do not create such scenarios from whole cloth; be 
realistic. And yet, let your imagination run a little, so that you have a 
good chance of thinking of the most dangerous possibilities. In particu- 
lar, think about the dangers of the implementation process, political and 
otherwise. Scenario writing also benefits from thinking about possible 


20. For further details, see Morgan and Henrion (1990, chap. 8). You can use the 
commercially available (and very user-friendly) Crystal Ball program to run Monte 
Carlo simulations. 

21. Most of the work on this type of simulation has been done at the RAND 
Corporation. See Lempert, Popper, and Bankes (2003). 
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failures from a vantage point in the future looking backward. Consider 
the following scenarios: 


e Ina health or safety regulatory program, the scientific or technical 
knowledge necessary to produce rational and legally defensible 
standards may prove to be lacking. As a result, five years from now, 
symbolic politics, corruption, industry capture, or excessive regula- 
tory zeal will have filled the vacuum. 

e Time passes, and budgetary resources and political support that 
were once available slip away under the impact of electoral change 
and shifts in the economy. A terrorist-identification program, 
begun under nurturant leaders and accompanied by editorialists’ 
applause, will have become consolidated with another program, 
then taken over by a different bureaucratic unit, and eventually will 
have disappeared. 

e A successful state program designed to furnish technical assistance 
to extremely poor rural counties will have added a mandate to aid 
many not-so-poor urban counties, with the result that scarce pro- 
gram resources will have been dissipated and squandered. (I call 
this scenario “piling on”; see Bardach 1977). 

e A program that subsidizes research and development of “fish pro- 
tein concentrate,” intended as a cheap and nutritious food additive, 
is launched with great fanfare. Five years from now it will have been 
stalled, permanently, by the U.S. Food and Drug Administration, 
which will not have been able to assimilate this product into its 
standard operating procedures for regulatory review. 


AA] Notice that these scenarios are written in the future per- 
fect tense. Use of this verb tense encourages concreteness, which is a 
helpful stimulant to the imagination (Weick 1979, 195-200). It often 
helps your scenario writing to start with a list of adverse implementa- 
tion outcomes, conjuring up one or more scenarios about how each of 
them might occur. Remember the list of such outcomes embodied in the 
scenarios just described: long delays; “capture” of program or policy 
benefits by a relatively undeserving and unintended constituency; exces- 
sive budgetary or administrative costs; scandal arising from fraud, waste, 
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and abuse that undermines political support and embarrasses support- 
ers; and administrative complexities that leave citizens (and program 
managers) uncertain as to what benefits are available or what regula- 
tions must be complied with. 


_ TEREST Undesirable side effects. Analysts are often cautioned to 
think about “unanticipated consequences.” But this term is not appropri- 
ate, for it is often used to refer to perfectly anticipatable, though undesir- 
able, side effects. Here are some common undesirable but foreseeable 
side effects in public programs: 


e Moral hazard increases. That is, your policy has the effect of insulat- 
ing people from the consequences of their actions. For example, 
increasing the size of unemployment benefits has the side effect of 
blunting the incentives to search for a replacement job. 

e Reasonable regulation drifts toward overregulation, especially if the 
costs of overregulation are not perceptible to those who bear them. 
One possible adverse result of setting health or safety standards 
“too high” and enforcing them “too uniformly” is that you increase 
private-sector costs beyond some optimal level. For instance, given 
most people’s preferences for safety, imposing auto bumper stan- 
dards that cost some $25 per vehicle, but have only trivial effects on 
improving vehicle crashworthiness, would not pass a conventional 
benefit-cost test. 

A second adverse result of overregulation might be that you inad- 
vertently cause a shift away from the regulated activity into some 
other activity that—perversely—is less safe or less healthful. For 
instance, some observers argue that overregulating the safety fea- 
tures of nuclear power production has caused a shift toward coal, 
which they argue is much more hazardous than nuclear power. 

e Rent-seekers—that is, interests looking out for profitable niches 
protected from full competition—distort the program to serve 
their own interests. It is not inevitable that suppliers of goods and 
services to the government, including civil servants, will find ways 
to capture “rents,” but it often happens (e.g., with many defense 
contractors). Rent-seeking also occurs in less obvious ways—as 
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when some regulated firms successfully lobby for regulations that 
impose much higher compliance costs on their competitors than 
on themselves. 


The ethical costs of optimism. It is hard to overstate the importance of 
worrying about the possible adverse side effects of otherwise “good” 
policies, not to mention the possibility that even intended main benefits 
may fail to materialize under many circumstances (see the chapter on 
“Assessing Your Ignorance” in Behn and Vaupel 1982). The ethical policy 
analyst always poses the question, “If people actually were to follow my 
advice, what might be the costs of my having been wrong, and who 
would have to bear them?” Keep in mind that the analyst typically is not 
one of the parties who have to bear the costs of her mistakes. 


The Emergent-Features Problem 


Policy often intervenes in systems of some complexity, systems populated 
by actors who adapt to your interventions in surprising ways and whose 
adaptations lead other actors to create still further adaptations. Surprising 
behavior may emerge from such dynamics. How can you take such pos- 
sibilities into account when you make your projections? 

In many cases, you cannot, for the systems are too complex and too 
little understood. The macro-economy is an extreme case—the hypo- 
thetical responses of producer interests to “supply-side” tax cuts are a 
major source of contention between those who think the taxes generated 
by a growing economy will substantially offset the direct effects of the 
cuts and those who are deeply skeptical of this scenario. Few cases are 
that extreme, however, and you might make some progress with what 
might be called “the other-guy’s-shoes” heuristic. 

Imagine yourself in the other guy’s shoes. Say to yourself: “If I were X, 
how would I act?” And then proceed to crawl into X’s mind and play out, 
in your own mind, what X might do. Do this systematically for each of 
the important stakeholders or other affected parties. The value of this 
exercise is that you will discover them to be adapting in surprising ways 
to the new policy situation you may be creating. 

For example, under chemical right-to-know laws, workers must be 
told what substances they have been exposed to, and they may examine 
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health records maintained by employers. If you were a worker, how 
might you use this law? Might you use the information to quit your pres- 
ent job? To demand a higher wage or more protective equipment? To sue 
your employer or put pressure on your union representative? 

And how would your union representative react to such pressure? 
Might this pressure make the representative’s job harder—or perhaps 
easier in some way? 

Now, suppose that you were an employer. Given what you expect your 
workers to do, you would face incentives to make adaptations or coun- 
termoves. Might you stop keeping all health records not explicitly 
required by law? Or continue keeping records but permit doctors to per- 
form only selected lab tests? And if you were a worker and saw your 
employer doing these things, what countermoves would you make? 

Not all the moves and countermoves of players wearing the other 
guy’s shoes will necessarily lead to trouble for the policy alternative you 
are evaluating. Many such adaptation sequences may prove to be help- 
ful, in the sense that they may help society to adjust to the changes set 
in motion by the new policy. At some point in the 1970s, the Federal 
Trade Commission (FTC) attacked the problem of retailers evading 
implied warranty obligations for defective products by selling install- 
ment debts to banks and other collectors that had no duty, under the 
so-called holder-in-due-course doctrine, to fix the product or to refrain 
from collecting on the installment debt. The FTC solution was, in effect, 
to abolish the protections of the holder-in-due-course doctrine. Banks 
complained that they did not want to go into the toaster repair business. 
But if you put yourself in the shoes of a bank manager suddenly obliged 
to become a toaster repairer, might you not have thought of contracting 
out your repair obligations to repair specialists, or perhaps arranging 
not to buy installment debts from retailers who you believed could not 
be relied upon to make good on their implied warranties? 


Construct an Outcomes Matrix 


The step of projecting outcomes leads you into a dense thicket of infor- 
mation. At some point along the way, you will probably need to stand 
back and assess complex and uncertain scenarios for perhaps two to five 
basic alternatives, combined with their principal variants. A convenient 
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way to get an overview of all this information is to display it in an out- 
comes matrix. The typical outcomes matrix format arrays your policy 
alternatives down the rows and your evaluative criteria across the col- 
umns. Each cell contains the projected outcome of the row alternative as 
assessed by reference to the column criterion. 

Table I-1 is an example. It appeared in a report by four of my students 
in 2008 that had been requested by the international environmental 
group ICLEI—Local Governments for Sustainability.2* They projected 
outcomes for eight alternatives (“scenarios,” in their usage) across five 
criteria (in three clusters).?3 I do not vouch for the accuracy of their pro- 
jections, though they tried the best they could to synthesize the diverse 
and sometimes contradictory research literature as it existed at the time. 
Of greater interest is their attempt to fill in the cells in a canonical matrix 
form. The alternatives are listed down the rows and the criteria across the 
columns. Three criteria are lumped together under the heading “Viability,” 
though if space had permitted, the students might have made a separate 
column for each. The analysis applies to a representative U.S. city called 
Anytown. Note that the matrix is labeled as a “Comparative Analysis.” 
Each projection is compared to a baseline projection for the year 2050, 
showing only the difference between the baseline projection and the esti- 
mated projection for the indicated alternative. Most cells contain a pro- 
jected range rather than a single point estimate. In Step Six, I come back 
to Table I-1 and discuss how this comparative setup facilitates confront- 
ing the trade-offs. 

An outcomes matrix at this stage of your work is a scratch-pad affair, 
useful for you and your team members and perhaps a friendly outsider 
or two. Its main function is to help you see what you have in hand and 
what you still need to learn about. A secondary function is to prepare to 
confront the trade-offs (see Step Six). If the matrix looks to you large 
and complicated, you may be encouraged to shrink it: conceptualize 
some alternatives as mere variants of more or less the same thing, get 
rid of alternatives that are obvious losers, and omit criteria that don’t 


22. See Cryan et al. 2008. 
23. They grouped their eight alternatives into five sub-groups, however, to simplify 
the analysis. 
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COMPARATIVE ANALYSIS 
Anytown, U.S.A 


O (Operational) 
E (Economic) 


(2050 Baseline: 5.5 Million Metric Tons CO,e) P (Political) 
(% reduction from (Cost per ton 
2050 CO, e baseline) CO,e abated) 
Policy Scenario Efficacy Cost-effectiveness Viability 
Mandate efficiency 6.9% to 8.8% —$130 to $5 O: High 
retrofits for homes E: Medium 
P: High 
Mandate efficiency 7.9% to 10.5% —$132 to -$30 O: High 
retrofits for E: Medium 
commercial buildings P: High 
Require zero-energy 4.1% to 5.6% —$132 to -$25 O: High 
capable homes E: High 
P: High 
Require zero-energy 6.5% to 8.9% —$120 to -$48 O: High 
capable commercial E: High 
buildings P: High 
High-density 2.4% —$1,333 to -$702 O: High 
residential E: High 
development P: Medium 
5 E m 
Incentives for one aloes is pu 
distributed PV P: High 
$20 carbon tax 11.3% $20 O: High 
E: Medium 
P: Low 
$50 carbon tax 20.6% $50 O: High 
E: Medium 
P: Low 


differentiate among alternatives (i.e., all the alternatives appear to do 


about as well or as poorly with respect to these criteria). The students 


who produced Table I-1 excluded three alternatives that they had origi- 
nally considered: a local cap-and-trade program, leveraging collective 
purchasing power in energy markets, and urban forestry. 
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You may find it useful to go through this exercise more than once, as 
your analysis evolves over time. (Table I-1 is the final version of several 
matrixes that the student group made.) 

A later version of such a matrix may also prove useful when you tell 
your story (see Step Eight). However, unless the matrix is very well 
designed and explained, it can impede the flow of your story rather than 
assist it. 


E] Here is a tip with a graphic dimension. Take advantage of 
the fact that being listed earlier (more leftward) in the matrix is usually 
taken to signify greater importance. Even if you are unsure how to 
weight criteria on some cardinal scale, with equal intervals assumed 
between all it points, you might feel better about an ordinal scale, 
requiring judgments only of “more than” and “less than.” Put what you 
think should be the weightier criteria in the more leftward columns. A 
common error that occurs in labeling the criteria columns in such a 
matrix is to fail to indicate what value is at stake and in what dimensions 
the measurement is being done. For instance, if you are assessing a 
rental subsidy program and you enter a plus sign in a column labeled 
“Landlord/tenant relations,” the reader may not know whether you 
think relations will become more harmonious, more confrontational, 
less dominated by landlords, less dominated by tenants, or something 
else. It is not sufficient that your surrounding text makes your intention 
clear; the matrix label itself must be informative. In many cases it helps 
to insert the term maximize or minimize in the criterion label. Table I-1 
is exemplary in almost all respects, except that the column labels do not 
include such words. It happens that the meaning is quite clear from the 
context, of course, but in the interests of “analytic hygiene” it would 
have been better to include them. 

If you cannot fill in the cell with a quantitatively expressed description 
of the projected outcome, you may have to settle for a verbal descriptor 
such as “very good” or a symbolic descriptor such as + or —. The opera- 
tive word here, though, is cannot. Quantification goes a long way toward 
making an analysis useful, and rough yet adequate quantification is easier 
than you might suppose. Remember, also, the heuristic of increasing or 
decreasing “the odds,” mentioned in “Step One: Define the Problem.” 
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In listing or stating criteria, speak in the declarative, not the interroga- 
tive. “How equitable is the final budget outcome?” is not a criterion; it is 
a question. “Maximize equity” is a criterion. 


How Projecting Outcomes Connects to Design Problems 


seemantic Tip The Eightfold Path language of choice—problems, alterna- 
tives, criteria, projections, trade-offs—does not fit design problems (see 
“Step Three: Construct the Alternatives”) without a little translation. The 
object of design is usually an organization or a program that is, more 
generally, a system of interrelated parts. The object of design is to make 
the system perform in a certain way. You can translate the problem to be 
solved as the performance objective to be achieved. Meeting this objec- 
tive is the primary criterion of success. Other criteria are distinctly sec- 
ondary. Meeting a variety of other performance requirements usually 
means satisfying other criteria of the “constraint” type (see the earlier 
discussion of linear programming). 

Once you have designed the best system you possibly can, though, that 
design becomes just another entry in the set of alternatives you are con- 
sidering. Some other alternative, perhaps proposed by a stakeholder 
interest, less customized but perhaps also less chancy, might be sitting out 
there ready to beat back a challenge from your own best effort at design. 
The choice framework applies nicely at that point. 

In the choice framework, we typically find ourselves working on a 
single problem. That problem can have more than one aspect, though 
probably not too many. But because design problems encompass a sys- 
tem of interrelated parts, you may have to treat each part separately— 
that is, as a separate problem. For instance, consider redesigning a state 
educational system’s approach to delivering the full menu of college- 
preparatory courses to all high school students. This task involves a set of 
related but differentiable problems or performance objectives involving, 
among other things: 


e The level of discretion allowed school districts in imposing college- 
preparatory graduation requirements 

e How districts might provide student support, and how the state 
might support the districts in their efforts to do so 
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e The rapidity with which the new requirements might be implemented 

e The nature and adequacy of the procedures for monitoring high 
school course content 

e The proficiency level expected of students in the relevant high 


school courses24 


STEP SIX: CONFRONT THE TRADE-OFFS 


It sometimes happens that one of the policy alternatives under consider- 
ation is expected to produce a better outcome than any of the other 
alternatives with regard to every single evaluative criterion. In that case— 
called “dominance”—there are no trade-offs among the alternatives. 
Usually, though, you are less fortunate, and you must clarify the trade- 
offs between outcomes associated with different policy options for the 
sake of your client or audience. 


Focus on Outcomes 


A common pitfall in confronting trade-offs is to think and speak of 
the trade-offs as being across alternatives rather than across projected 
outcomes—for example, “trading off twenty foot-patrol police officers in 
the late night hours against a lower-maintenance-cost fleet of police 
vehicles.” Although such a trade-off exists, with a second’s thought you'll 
see that you can’t do anything at all with it. Both alternatives must first 
be converted into outcomes before genuine trade-offs can be confronted. 
Thus the competing outcomes might be fifty (plus or minus) burglaries 
per year prevented by the foot-patrol officers versus a savings of $300,000 
in fleet maintenance. 

The most common trade-off is between money and a good or service 
received by some proportion of the citizenry, such as extending library 
hours from 8 p.m. till 10 p.m., weighed against a cost of $200,000 annu- 
ally. Another common trade-off, especially in regulatory policies, involves 
weighing privately borne costs (a company’s installing pollution abate- 
ment equipment) against social benefits (improved health of the affected 
population and the protection of forests). If the projected outcomes can 
be monetized—that is, expressed in dollar terms—it is sometimes simple 


24. See Sutton (2008). 
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to evaluate the trade-offs. Just choose the option that yields the largest 
net value, once costs have been subtracted from benefits. This procedure 
applies nicely if budgets, and therefore the scope of the activity, are not 
limited. But it can run afoul of another monetary consideration, cost- 
effectiveness per unit of activity, if budgets or other inputs are limited. In 
Table I-1, note that the high-density residential development option— 
presumably limited in extent because of the limited likely scope of new 
development—is more cost-effective than any other activity but is less 
efficacious than all the others, too. 

In Table I-1, we see that there is no dominant outcome. The really 
efficacious options, involving a carbon tax, are not viable politically. 
Retrofitting existing buildings is apparently more efficacious than meet- 
ing green standards in new buildings, but it is a strategy deemed by the 
student-authors to be less viable than the latter. And, as we said earlier, 
making new developments denser, although cost-effective, is not very 
efficacious. 

The student-authors did not recommend choosing among these alter- 
natives, however, but recommended doing as many of them at one time as 
was feasible (“viable”). The trade-offs analysis would nonetheless permit 
decision makers, and the public, to prioritize which alternatives to empha- 
size in the likely case that priorities needed to be set. In their view, setting 
standards for energy efficiency in buildings was the first order of business. 


Establish Commensurability 


Suppose some Alternative A, stacks up very well on Criterion C,, mod- 
erately well on C,, and poorly on C,. And suppose that A, stacks up in 
the opposite way. We can choose between the two alternatives only if we 
can weight the importance of the criteria and if we can express their 
relative weights in units that are commensurable across the criteria. As 
you may have heard, money is everybody’s favorite candidate for the 
commensurable metric. Using money as the metric is a very good idea, 
and it often works much better than you might imagine. For instance, 
even the “value” of life can sometimes be described reasonably well in the 
metric “willingness to pay X dollars for a reduction in the risk of death 
by Y percent a year,” or something like it. 
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However, there are limits to the money metric and to commensurabil- 
ity, as well. To reach a summary judgment as to how much political equal- 
ity to give up in a political redistricting case, for instance, in exchange for 
how much more African American voter power, it seems impossible even 
to state the trade-off in meaningful terms. In general, this problem is 
known as the “multi-attribute problem.” In some deep sense the problem 
is logically insoluble, although some heuristics are available to help trim it 
down to its irreducible size.” 


Break-even analysis revisited. We have seen how break-even analysis can 
help you both to focus on which residual uncertainties you will have to 
estimate and to frame the terms in which those estimates must be given 
(e.g., “We have to believe Alternative A, will produce at least X results in 
order to justify choosing it.”). We turn now to how break-even analysis 
can also help to solve commensurability problems. 

Consider those policy areas, such as safety regulation, where we are 
often implicitly trading off dollars against risks to life. It might be sup- 
posed that in order to assess these proposals, you would have to decide 
what a human life is really worth—a task many of us, quite understand- 
ably, are unwilling to perform. The task is made somewhat more tractable, 
however, if you work with quantitative estimates and apply break-even 
analysis. Suppose, for instance, that you are considering whether or not 
to impose on the auto industry a new design standard that will 
improve safety and save an estimated twenty-five lives every year into the 
indefinite future. The cost of meeting the standard is estimated at $50 
million per year indefinitely. The trade-off at the margin appears to be, 
therefore, “$2 million per life.” But you don’t have to answer the question 
“What’s a human life really worth?” in order to make at least some sense 
of this decision. You do have to answer the question “Is a statistical life 


25. See Stokey and Zeckhauser (1978, 117—133) and MacRae and Whittington (1997, 
201-203). One potentially misleading heuristic has the analyst creating a score for 
each alternative with respect to each criterion and then manipulating the scores 
arithmetically. It is easy to get the arithmetic right, but it is often hard to come up 
with scoring procedures that are not at some level arbitrary (e.g., anchored against 
some arbitrarily defined level of excellence or its opposite). 
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(that is, the life of an unknown individual ‘drawn in a random manner 
from some population, rather than a named person’s life) worth at least 
$2 million?” That is a break-even analysis sort of question. For reasons 
best known to yourself, it may be obvious to you that a statistical life 
surely is—or isn’t—worth that much. And although it’s very difficult to 
decide whether the worth of a statistical life falls on one or the other side 
of some monetary boundary, it’s a lot less difficult than coming up with 
a point value. 

Even this sort of trade-off calculation is troubling to many people, and 
some find it morally repugnant. Unfortunately, repugnant or not, it is in 
a sense inevitable. Whatever position you take on the auto safety design 
standard described, you are by implication also taking a position on the 
dollars/risk-to-life trade-off: if you favor the standard, you implicitly 
believe the trade-off is worthwhile, whereas if you oppose it, you don’t. 
Fortunately, this logical implication has its uses. You may in many cir- 
cumstances quite sensibly prefer to rely on your intuition rather than on 
some complicated systematic method. Once you have reached your con- 
clusion on that basis, though, you should check your intuition by asking 
yourself: “Since the implication of my policy choice is that I value X as 
being worth at least (or at most) thus-and-such, do I really believe that?” 


Choosing a Base Case 


Choosing a base case against which to compare the other alternatives is 
unavoidable. Even if the base case implicitly is “whatever situation exists 
today,” which is the simplest and most obvious option, this does repre- 
sent a choice. What are some of the other possible choices? Here is a list 
of some possibilities, along with a brief commentary: 


e Future conditions provided that business were to continue as usual. 
This is what the authors of the matrix in Table I-1 chose. But what 
is included in the “business as usual” base case? The authors of 
Table I-1 assumed no new regulations and no changes in fossil fuel 
consumption other than those caused by demographic changes. 
They did not include possible technological changes, for instance. 

e Changes from the present that would occur if some policy not in the 
matrix were to be adopted. Suppose, for instance, that the state were 
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likely to finance and construct a train system connecting major cit- 
ies, and that this system were expected to reduce automobile usage 
overall by, say, 5 percent. This is like “business-as-usual” except that 
changes caused by a particular policy are in sharper focus. 

e Projections of the results of one particular policy option. In 1996 the 
RAND Corporation, referred to earlier, published a study compar- 
ing the cost-effectiveness of crime reduction strategies programs to 
the base case of the “three-strikes” mandatory incarceration policy 
that California had recently adopted. Three out of the four pro- 
grams were clearly more cost-effective, and the fourth possibly so. 
The objective was to show that on narrow crime-prevention grounds 
alone, and leaving aside humanistic considerations, “three-strikes” 
was wasting taxpayer money, since other options were cheaper for 
achieving the same objective. 


What difference does it make which base case is used? In some cases, 
it definitely helps to focus attention on the worth of the base case itself, 
particularly if, as in the three-strikes case, it is controversial. In other 
situations, it simply provides a way of standardizing the background 
against which trade-offs are evaluated across the other alternatives in the 
matrix, as happens in the matrix in Table I-1. In effect, we ask how a 
variety of alternatives stack up when it comes to improving on the situa- 
tion described by the base case.”° 

Once the base case is chosen, and the cells are filled in, and commen- 
surability is pushed as far as can be done, what next? As economics 
teaches us, trade-offs occur at the margin. Trade-off analysis tells us 
something like this: “If we spend an extra X dollars for an extra unit of 


26. When the base case is not the current situation but some array of hypothetical 
outcomes, it may contain errors. In using comparisons to the base case simply in 
order to compare the worth of other alternatives, these errors would be inconsequen- 
tial if each of the alternatives were to be equally affected by the errors. But this will not 
always be true. For instance, suppose the federal government were to impose a $10/ton 
carbon tax in, say, 2025. It might matter, in comparing the alternatives in Table I-1 
if, as a result, owners of existing buildings were to retrofit, and were to work on their 
least energy-efficient buildings first, while new construction were less affected. In that 
case, the first two options in Table I-1 would look a lot less attractive relative to the 
other alternatives, since relatively severe mandates would not be necessary. 
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Service Y, we can get an extra Z units of good outcome.” This kind of 
analysis puts the decision maker in the position to answer the question 
“Does society (or do you) value Z more or less than X?” and then to fol- 
low the obvious implication of the answer: if yes, decide for another unit 
of Y; if no, don’t. 


meemantic Tip IN linguistic device to help you stay focused on the margin 
is frequent use of the word extra. Note that this word appears three times 
in the example analysis in the preceding paragraph. 

Some units of Service Y can be purchased only in “lumps” larger than 
one—sometimes much larger. Consider transportation services provided 
by highways and bridges. Y might be one passenger trip from A to B, but 
most transportation construction projects (highway enlargements, new 
bridge crossings) can be undertaken only for minimum bundles of Y that 
run into the thousands of trips. Or suppose that a police chief must 
choose one of two “lumpy” alternatives, such as $1 million per year for 
more overtime on the night shift or $250,000 per year for more rapid 
replacement of police cars. The first alternative is lumpy because the 
police union insists on a minimum overtime rate for all 150 officers on 
the shift, and the second is lumpy because the auto supplier charges 
much less per vehicle after some threshold number of vehicles. If, say, the 
projected decrease in burglaries from increased overtime were 200 per 
year and that from newer vehicles were 50, the trade-off confronting the 
decision maker at the margin is an extra $5,000 per extra burglary pre- 
vented. In this case the margin is a lumpy 150 burglaries and $750,000. 
(Criteria other than burglary prevention and cost-efficiency would, of 
course, be relevant to this problem.) 


Stop! Focus, Narrow, Deepen! 


Up to this point, progress on the Eightfold Path has mainly bred expan- 
sion: of problem elements, alternatives, and criteria. It may also have bred 
an undesirable formalism, such that lists of these items may have come 
to have a life of their own. The outcomes matrix, which ideally would 
have served as a sort of “rough draft with attitude,” may have displaced 
the problem with which the project began. But the object of all your 
analytic effort should not be merely to present the client with a list of 
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well-worked-out options. It should be to ensure that at least one of 
them—and more than one, if possible—would be an excellent choice to 
take aim at solving, or mitigating, the problem. 

At a minimum, this need to focus, narrow, and deepen your analysis 
of the most promising alternative(s) means that you must think very 
seriously about (1) the politics of getting this alternative legitimated and 
adopted, and (2) the design of the ongoing institutional features that will 
have the power and resources to implement the policy or program in the 


long run.?7 


STEP SEVEN: DECIDE! 


This step appears in the Eightfold Path as a check on how well you have 
done your work up to this point. Even though you personally may not be 
the decision maker, you should at this point pretend that you are. Then, 
decide what to do, based on your own analysis. If you find this decision 
difficult or troublesome, the reason may be that you have not clarified 
the trade-offs sufficiently, or that you have not thought quite enough 
about the probability of serious implementation problems emerging (or 
not emerging), or that a crucial cost estimate is still too fuzzy and uncer- 
tain, or that you have not approximated carefully enough the elasticity of 
some important demand curve, and so on. 

Think of it this way: unless you can convince yourself of the plausibil- 
ity of some course of action, you probably won't be able to convince your 
client—and rightly so. 

Of course, when you tell your story to your client or any other audi- 
ence, you may not think it appropriate to make reference to your own 
decision. You may choose, instead, to simply limit your story to a clarifi- 
cation of the relevant trade-offs and leave the decision completely up to 
the audience. 


Apply the Twenty-Dollar-Bill Test 


You should at this point subject your favored policy alternative to the 
“twenty-dollar-bill test.” The name of this test is based on an old joke that 


27. For reasons of space, I do not discuss the first of these matters here, but see 
Appendix C for a very brief survey of pertinent institutional issues. 


70 A PRACTICAL GUIDE FOR POLICY ANALYSIS 


makes fun of economists. Two friends are walking down the street when 
one stops to pick something up. “What about that—a twenty-dollar bill!” 
he says. “Couldn't be,” says the other, an economist. “If it were, somebody 
would have picked it up already.” The analogy is this: if your favorite 
policy alternative is such a great idea, how come it’s not happening already? 
The most common sources of failure on this test are neglecting to con- 
sider the resistance of bureaucratic and other stakeholders in the status 
quo, and the lack of an entrepreneur in the relevant policy environment 
who has the incentive to pick up what seems like a great idea and see it 
through. Failure on this test is not fatal, of course. Just keep fiddling until 
you invent a variant of your basic idea that will pass. 


STEP EIGHT: TELL YOUR STORY 


After many iterations of some or all of the steps recommended here— 
principally, redefining your problem, reconceptualizing your alternatives, 
reconsidering your criteria, reassessing your projections, reevaluating the 
trade-offs—you are ready to tell your story to some audience. The audi- 
ence may be your client, or it may include a broader aggregation of stake- 
holders and interested parties. It may be hostile, or it may be friendly. 
Your presentation may be a one-time-only telling, or it may be merely the 
first effort in a planned long-term campaign to gather support behind a 
legislative or executive change. (For a discussion of the issues likely to be 
involved in such a campaign, see Appendix D, “Strategic Advice on the 
Dynamics of Political Support.”) 


Apply the Grandma Bessie Test 


Before proceeding further, however, you need another little reality check. 
Suppose your Grandma Bessie, who is intelligent but not very sophisti- 
cated politically, asks you about your work. You say you are a “policy 
analyst working for. .. .” She says, “What’s that?” You explain that you’ve 
been working on “the problem of. . . .” She says, “So, what’s the answer?” 
You have one minute to offer a coherent, down-to-earth explanation 
before her eyes glaze over. If you feel yourself starting to hem and haw, 
you haven't really understood your own conclusions at a deep enough 
level to make sense to others, and probably not to yourself, either. Back 
to the drawing board until you get it straight. 


THE EIGHTFOLD PATH 71 


Now consider the possibility that someone might actually wish to 
base a real decision or a policy proposal on your analysis. (It’s been 
known to happen.) Even if you, as an analyst, would not have to deal 
directly with such a tough audience as Grandma Bessie and her kin 
(including, of course, Grandpa Max), it’s likely that someone will have 
to do so. At the very least, therefore, you'll have to be able to explain your 
basic story to someone in sufficiently simple and down-to-earth terms 
that that someone will be able to carry on with the task of public, dem- 


ocratic education.”® 


Gauge Your Audience(s) 


Assuming that you've passed the Grandma Bessie test, identify and assess 
the likely audience(s) that are more sophisticated and involved than 
Grandma Bessie. 

First comes your client, the person or persons whose approval you 
need most—your hierarchical superior(s), perhaps, or those who are 
funding your work. What is the relationship between you and your cli- 
ent? What you say and how you say it should depend a great deal on 
whether your relationship is long term and on whether it is carried on 
face to face. In particular, how easy will it be for you to correct any mis- 
understandings that may arise? 

Next, think about the larger political environment. Who do you think 
will “use” the analysis and for what purpose(s)? Will anyone pick up your 
results for use in an advocacy context? Would you regard this use of your 
results as desirable? Desirable if certain advocates use your work and 
undesirable if others do so? Do you want to do anything to segregate the 
elements of your analysis by the type of audience you might want it to 
reach—or not reach? Are you, perhaps inadvertently, using scare words 
that will alienate certain audiences? 


28. Sometimes this is referred to as the challenge of giving an “elevator speech.” You 
and your boss, or some relevant other, find yourselves together in an elevator for too 
long a time to make do with just “Hi, how are ya?” The boss asks how your project 
is going. You have maybe a minute to explain what you're up to and why he should 
be interested and perhaps persuaded. So have your “elevator speech” committed to 
memory and ready to go at a moment’s notice. 
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If you are making a clear recommendation, make sure that you raise 
and rebut possible objections to it that might occur to various important 
audiences. Also make sure that you compare it to what you or others 
might regard as the next best course of action, so as to be ready to show 
why yours is better. 


Consider What Medium to Use 


You can tell your story in written or in oral form. In either case, com- 
municate simply and clearly. The guiding principle is that, other things 
being equal, shorter is always better. In written presentations, good sub- 
headings and graphics can make reading and comprehension easier. 
Visual aids such as flip charts, overhead transparencies, and computer- 
based slide projections often help in oral presentations. 

Oral presentations require practice, self-discipline, and a little knowl- 
edge of some basic principles. The most basic of the basic principles are 
these: speak very slowly and distinctly; speak loudly enough to be heard 
throughout the room, even over distracting noises; speak in a lower reg- 
ister, which tends to increase perceived trustworthiness and credibility; 
do not fidget, but don’t stand like a stick, either; make lots of eye contact 
with audience members and, in doing so, don’t favor one side of the 
room over another. Speaking slowly and distinctly is probably harder 
than you think—and more important, too. 


Give Your Story a Logical Narrative Flow 


Your story’s flow should be designed with the reader’s (or listener’s) 
needs and interests and abilities in mind. In both written and oral pre- 
sentations, it should be evident to the audience what motivates the entire 
analysis. Therefore, it is best to open with a statement of the problem 
your analysis addresses.? 

It is also important to motivate the more detailed steps in the flow of 
the analysis, that is, the sections, paragraphs, and sentences. Most readers 


will look for the motivation of any element in what immediately precedes 


29. An unusually fine manual on how to give slide-based oral briefings is published 
by the RAND Corporation (1994). 
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it, which makes it important to avoid lengthy digressions. For these rea- 
sons, be wary of sections that you are tempted to label “Background.” 
Similarly, the phrases “Before turning to ...” and “It is first necessary to 
explain/understand the history of .. ? are usually signs of undigested 
material. Many readers will be alert to these danger signs, so you should be, 
too. Policy analysis, remember, is about the future. It is often not obvious 
how or whether history affects the future, and the burden should be on the 
writer or speaker to show exactly how this effect will come about. 

A common, though not uniformly applicable, organizing framework 
is to begin with a good problem definition and then to treat each alterna- 
tive you consider as a major section. Within each such section, you proj- 
ect the probable outcome(s) of implementing the alternative and assess 
how likely such outcome(s) are in the light of some causal model and 
associated evidence. Following these discussions, you review and sum- 
marize the alternative outcomes and discuss their trade-offs. This frame- 
work contains no special discussion of criteria; however, sometimes an 
explicit discussion of criteria is important. It might appear either just 
before or just after the presentation of the alternatives and their associ- 
ated outcomes. 


Some Common Pitfalls 


Following the Eightfold Path too closely. Sometimes it helps to structure 
your narrative flow as though you were leading the reader by the hand 
down the Eightfold Path. But usually this approach is a mistake. The 
purpose of the Eightfold Path, remember, is to help you think through a 
complicated problem. It is not at all necessary to use it in telling the story, 
though some aspects of it sometimes help. 


Compulsive qualifying. Don’t interrupt the flow of an argument in order 
to display all the qualifications and uncertainties about some particular 
element in the argument. A linguistic way around this pitfall is to use 
adjectives or adjective phrases such as most, on average, and more often 
than not to state the generality, and then to return to the exceptions in the 
next section. (Or, if the exceptions and qualifications really can’t wait, try 
a parenthetical sentence or a footnote.) 
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Showing off all your work. Don’t include every fact you ever learned in 
the course of your research. Even if youve done a good and thorough job 
of research and analysis, most of what you have learned will prove to be 
irrelevant by the time you're finished. That is, you will have succeeded in 
focusing your own attention on what is really important and in down- 
playing what only appeared important at the beginning. You don’t usu- 
ally need to take your reader on the same wandering course you were 
obliged to follow. 


Listing without explaining. Should you list every alternative policy that 
you intend to analyze in the report before you actually get around to 
providing the analysis? Such a list is a good thing when the alternatives 
are not numerous, when they are all taken seriously either by you or by 
your audience, and when they will prepare the reader’s mind for the 
detailed assessment that will follow. However, if you have many alterna- 
tives to consider, the reader will forget what’s on the list, and if some of 
the alternatives turn out to be easily dismissed upon closer scrutiny, 
you'll simply have been setting up straw men and wasting the reader’s 
mental energy. 

Similarly, be cautious about listing every evaluative criterion of inter- 
est before coming to the assessment of the alternatives being considered. 
Usually—though not always—there is not much to be said in a separate 
section about criteria that can’t be better said when youre actually writ- 
ing the assessment sections. 


Spinning a mystery yarn. Start with the conclusion, the bottom line, the 
absolutely most interesting point you intend to make. Then present all 
the reasoning and evidence that you have to make your audience reach 
the same conclusions you have reached. In short, follow the opposite 
strategy from that which a novelist would follow. 


Inflating the style. Avoid the pomposity and circumlocutions of the 
bureaucratic and the academic styles. (Essential reading: George 
Orwell, Politics and the English Language.) Also to be avoided: a chatty, 
insider’s style—such as, “We all understand what creeps our opponents 
are, don’t we?” 


THE EIGHTFOLD PATH 75 


Structure Your Report 


Unless the report is short, begin with an executive summary. 

If your report is over fifteen to twenty pages long, say, a table of con- 
tents may well be helpful. If there are many tables and figures, either in 
the text or in the appendixes, a list of these items can be helpful, as well. 
Detailed technical information or calculations should appear in appen- 
dixes rather than in the text. However, enough technical information, 
and reasoning, should appear in the text itself to persuade the reader that 
you really do know what you're talking about and that your argument is 
at least credible. 

Use headings and subheadings to keep the reader oriented and to 
break up large bodies of text; make sure your formatting (capital letters, 
italics, boldface, indentation) is compatible with, and indeed supports, 
the logical hierarchy of your argument. 


Table format. Current professional practice is very poor with respect to 
the formatting of tables. Do not imitate it but strive to improve it. Every 
table (or figure) should have a number (Table 1, for instance, or Figure 3-A) 
and a title. The title should be intelligible; it is often useful to have the 
title describe the main point to be learned from the table (e.g., “Actual 
Risks of Drinking and Driving Rise Rapidly with Number of Drinks— 
but Are Greatly Underestimated by College Students”). Each row and 
column in a table must be labeled, and the label should be interpretable 
without too much difficulty. 

Normally, a table either is purely descriptive or is designed to demon- 
strate some causal relationship. In the latter case, it is usually desirable to 
create a table that makes a single point (or at most two) and that can stand 
alone without need of much explanation in the surrounding text. It is 
usually better to use two or three small tables to make two or three points 
than to construct one massive table and then try to explain its contents by 
means of the text that surrounds it. 

Tables usually require footnotes, and there should almost always be a 
source note at the bottom. Sometimes these notes refer to data sources 
used to make the table, and sometimes they attempt to clarify the mean- 
ing of the row or column labels, which are necessarily abbreviated. 
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Please do NOT imitate academic practice, which is to overstuff tables 
with all kinds of numbers and to mindlessly apply obscure column and 
row labels. Academic practice presupposes that all the data have been 
gathered “scientifically” and without serious bias; therefore, the presenta- 
tion style aims to convey these facts. Unimportant data share space in 
academic tables with important data so as to permit the reader to see that 
the complete truth has been told, that the author has not cherry-picked 
the data to convey only what is interesting and has conveyed the full story 
about what is statistically significant as well as what is not. If these issues 
are important to you and your readers, by all means provide the full 
story. But do it in appendixes. In most cases, though, try to minimize the 
information provided in a single table. 


References and sources. Include a listing of references and sources at the 
end of the presentation. Books and articles should be cited in academic 
style (alphabetical order by author). The main point is to provide biblio- 
graphic help to curious or skeptical readers who want to track down 
references for themselves. There are several acceptable styles, but a good 
model is the one used in the book review section of the Journal of Policy 
Analysis and Management, which is simple and direct. 

The current trend is toward “scientific citation” in lieu of footnote 
references in the text. That is, cite the author’s last name and year of pub- 
lication in parentheses in the text; the reader then consults the references 
section at the end for the full citation. If you follow this practice, the 
reference section should list the author(s) and year before the title of the 
work and other publication details. Sometimes you will want to include 
a page number in the parenthetical citation, as well. 

Legal citation style is quite different. If most of the references are legal, 
then it is advisable to cite all references in bottom-of-page footnotes. 
However, you can keep the scientific citation format within the footnotes. 

Notes are easier to read if they appear on the same page as the referenced 
text—that is, if you display them as footnotes rather than as endnotes. 


Using a Memo Format 


If your analysis is to be delivered in a memo, you should present it within 
a standard memo format, as follows: 
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[Date] 
To: [Recipient name(s), official position(s) ] 


From: [Your name, position. Sign or initial next to or above your 
name. ] 


Subject: [Brief and grammatically correct description of the 
subject] 


[The first sentence or two should remind recipient of the fact 
that she or he asked you for a memo on this subject, and why. 
Alternatively, you could explain why you are submitting this 
memo on this subject to the recipient at this time. ] 


[If the memo is long, you might open and close with a sum- 
mary paragraph or two. If you open with a long summary, the 
closing summary can be short. ] 


[If the memo is long, consider breaking it up with subheads. | 


Develop a Press Release 


Most policy analyses do not become the subjects of press releases or of 
radio or television sound bites, but some do. Others become candidates 
for such treatment, and all can profit, even in their extended form, from 
the analyst’s reflecting on how to condense the essential message. 
Hence, it will probably serve an analytic purpose—and sometimes a 
political one—if you sketch out a press release or a few ideas for sound 
bites. You may also want to think strategically and defensively to see 
how an opponent might characterize your work in a press release or 
sound bite. 


PowerPoint 


PowerPoint slides often supplement oral presentations, and indeed 
sometimes replace written reports altogether as nonverbal means of 
communication. The latter practice does economize on staff time to 
some extent, but it has the drawback (in my eyes, at least) of forcing the 
reader to imagine the connective issue that oral presentations normally 
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provides. In any case, my comments on PowerPoint as supplementation 
will be brief and rather personal, as plenty of full-scale manuals are 
available. 


e Keep it simple: have each slide present a separate point; use phrases, 
not sentences; and use only two or at most three colors. 

e Avoid cutesy icons and “cool” moving animals. 

e Think of the viewer’s needs: to see letters and numbers at a fair 
distance, and to not be bored by having you, as presenter, simply 
read what is on the screen. 

e Display the slide for long enough so that the viewer could actually 
read and absorb its contents—especially important for tables and 
graphs. 

e Include slides at suitable intervals that summarize what has been 
said so far and point the way to what is yet to come. 

e Make available to the audience, after the presentation—not during, 
as it is distracting—hard copies of slides (arranged six per page). 

e Visual supplements, such as photographs, can nicely support all the 
words, provided they are carefully chosen and displayed. 


PART 


ASSEMBLING EVIDENCE 


C onsider the problems confronting you as a researcher preparing an 
analysis of water pollution control programs for Blue Lake. You 
know that there is a dirty lake; that federal, state, and local legislation is 
directed toward the goal of cleaning up the lake (or preventing it from 
getting much dirtier); and that a state environmental protection office in 
the area has something to do with administering some or all of the rel- 
evant antipollution policies or programs. But you need to know more. 
You need to map the present policies and programs, their political envi- 
ronment, the ways in which the bureaucracies function to implement 
them, and the criteria by which experts and nonprofessionals evaluate 
them. You also need to make some decisions about how you will evalu- 
ate them. Then you need to learn what data are relevant to these criteria 
and figure out how to obtain these data. If you are planning to recom- 
mend changes in existing programs, you must develop the evidence that 
will permit you to make reasonable projections of the likely outcomes. 
In addition, you must learn what sort of changes the present set of rel- 
evant actors may be prepared to make or are capable of making. 

These are large challenges, but your resources in time, energy, money, 
and the goodwill of potential informants and interviewers are probably 
not at all large. Moreover, you would like to finish the study in no more 
than six months, let us say, and you do not want to waste the first five 
months simply getting your bearings. Where are you to begin? And hav- 
ing begun, how are you to proceed efficiently? 
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GETTING STARTED 


The first step is simple: Start with what you know. This injunction may seem 
self-evident or trivial or both. In fact, it is common for people to act in 
contradiction of it. Confronted by a new and challenging research task, 
they expect to flounder anxiously for a few weeks or months. And, behold, 
they do, for feeling stupid makes you so. Rarely is this waste of time and 
energy necessary, however. A few facts, or even vague recollections, plus 
some intelligent reasoning can usually move the project onto firm footing 
surprisingly quickly. Suppose, for example, that you are asked to do a policy 
analysis of “the future of the Wichahissic bituminous coal industry,’ a sub- 
ject as remote from your interest or previous experience as galactic spectros- 
copy. You might take stock by writing a memo to yourself as follows: 


e I was probably asked to do this study because someone thinks the 
future of the Wichahissic bituminous coal industry is pretty bleak 
or else because it is looking up. If the former, the results will prob- 
ably be used to justify some sort of government subsidy; if the 
latter, the results will be used for promotional purposes by the 
industry itself or by local merchants whose livelihood depends on 
the health of the industry. 

e The future of any industry depends in part on market demand. The 
demand for coal has probably been declining, partly due to the 
availability of substitute fuels. 

e Maybe high production costs imperil the health of the industry. 
Could it be that coal-mining technology is underdeveloped? If so, 
why? Perhaps the coalfields are running out and the technology has 
not been developed to handle poor, as opposed to rich, deposits. 

e There were a lot of miners’ strikes a few years ago. Are labor- 
management relations better or worse now? Are wage demands 
forcing the companies to go under? 

e Coal transportation depends on railroads. So, if the railroads are 
sick, could coal be well? 

e Coal is black and sooty, gives off a lot of smoke and has a nasty 
carbon footprint. Surely this is an ecological menace. Who, if any- 
one, is paying attention to this problem? Or is it really a problem? 
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Coal mining destroys the beauty, and probably the ecology, of the 
countryside. Is this really so? Might the Sierra Club have useful data 
on these questions? 

e Perhaps coal is not sick, just bituminous coal. Maybe the anthracite 
industry is flourishing. Surely there is a trade association of coal- 
mining companies with data here. Call up the nearest big coal-mining 
company and find out its name and address from the public relations 
office. 

e Perhaps coal is okay, but Wichahissic has a problem. But then again, 
Wichahissic does not seem to be as much in the news as Pokanoka, 
whose plight seems to be the archetype for “the depressed area.” Check 
BLS (Bureau of Labor Statistics) for unemployment figures here. 


Writing memos of this kind to yourself is useful not only at the begin- 
ning of a project but whenever you feel yourself beginning to drift toward 
panic or confusion. Following this initial stock-taking, you should think 
of yourself as designing, executing, and periodically readjusting a research 
strategy that will exploit certain predictable changes in your potential for 
gaining and utilizing information: 


e Locating relevant sources. Over time, you decrease your uncertainty 
about what is worth knowing and how to learn it. 

e Gaining and maintaining access to sources. (1) Over time, you aug- 
ment your ability to arrange interviews with busy or hostile per- 
sons, and to obtain data that are not clearly in the public domain; 
(2) over time, you also—and unavoidably—use up your access to 
certain sources, and you must therefore conserve such exhaustible 
resources for use only when the time is propitious. 

e Accumulating background information as leverage. Over time, you 
improve your capacity to interpret data and to force them out of 
reluctant sources, thereby increasing your background knowledge. 

e Protecting political credibility. Over time, the research process itself 
creates an environment that will either help or hinder the adop- 
tion and implementation of your—or your client’s—eventual 
recommendations. 
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The optimal strategy for managing any of these problems may conflict 
with the optimal strategies for dealing with the others. Therefore, after each 
problem is discussed in a separate section below, the final section of this 
chapter is reserved for a brief treatment of the trade-offs involved in trying 
to meet all strategic imperatives simultaneously. I assume throughout that 
the reader is an inexperienced policy researcher who has had academic 
training in the social sciences. Hence, I go to some lengths, at various 
points, to allude to differences between social science research methodol- 
ogy and the methods of policy research. I trust that the more experienced 
researcher will also find some profit in the arguments here, if only to con- 
ceptualize more clearly what she has already learned to do intuitively. 

A further clarification about the intended audience is in order. You 
start your task with certain resources and constraints, some of which are 
derived from your own experience and personality and others from your 
institutional location. Although institutional location is especially impor- 
tant in designing an optimal research strategy, it is not discussed in this 
book. Suffice it to say that the resources and constraints of a legislative 
staff assistant are quite different from those of her counterpart in a 
bureaucratic setting and are even more dissimilar to those of a student 
working with a campus-based Public Interest Research Group (PIRG). 
The strategic advice offered here is intended to be sufficiently general to 
meet the needs of researchers in any of these circumstances, however. 


LOCATING RELEVANT SOURCES 


Unlike most social science research, most policy research is derivative 
rather than original. That is, it is produced by creative play with ideas and 
data already developed by others. Only occasionally does the policy 
researcher set out to generate new data or assume responsibility for 
inventing a bright policy idea from scratch. Instead, the researcher’s role 
is preeminently discovering, collating, interpreting, criticizing, and syn- 
thesizing ideas and data that others have developed already. To be sure, 
social science research often works this way, too, but it also places a much 
higher premium on originality. In a sense, the policy researcher becomes 
an expert on experts—those scholars and persons of experience who are 
thought to be relatively sophisticated about the policy area. 


ASSEMBLING EVIDENCE 83 


Consulting Both Documents and People 


In policy research, almost all likely sources of information, data, and ideas 
fall into two general types: documents and people. By documents I mean 
anything that has to be read: Web sites, journal articles, books, newspapers 
and magazines, government reports, statistical archives, interoffice mem- 
oranda, position papers, bulletins, and so on. By people I mean anyone, 
whether a single individual or a group, who is to be consulted in person. 
Research on any policy problem usually entails a canvass of both types 
of sources. 

Avoid the pitfall of overemphasizing one type at the expense of the 
other. Sometimes you fall into the trap out of habit: if you start out 
interviewing experts, experienced administrators, and other informed 
persons, you continue doing so until you come to define “interviewing” 
as what your job is all about. You forget that the experts themselves 
typically have obtained a good deal of their expertise by studying docu- 
ments, and that much of what administrators offer can also be found 
in agency reports, legislative hearings, published statutes and regula- 
tions, and so on. 

Another reason for getting stuck in one medium and neglecting the 
other is an individual preference for less or more personal interaction— 
that is, for choosing to conduct your research via the Internet or in librar- 
ies (or files, in an organizational setting) or for concentrating on field- 
work instead. But it is usually desirable not only to consult both types of 
sources (documents and people) but also to consult them in alternating 
order: a spate of interviewing followed by a retreat to the Internet or the 
library followed by another round of interviewing, and so on. If for no 
other reason, there is probably a psychic economy in arranging and exe- 
cuting a fieldwork agenda in a consolidated time span, as there is in col- 
lecting and exploring a large body of documentary material. 

In a more general way, however, one source should be used to locate 
another and this branching out can just as easily lead from one medium 
to the other as it can from source to source of the same type. More explic- 
itly: people lead to documents as well as to other people, and documents 
lead to people as well as to other documents. There are thus four basic 
branches on the tree of knowledge, each of which I discuss in turn. 
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People leading to people. Often one informant leads spontaneously to 
another by remarking during the course of an interview or a conversa- 
tion, “Have you seen X yet? She’s very knowledgeable about. . . .” This 
information can be stimulated by your asking questions such as “Who 
else would be a good person to talk to about. .. .?” or the more specific 
“Who would be a good person to see in Agency Y?” For reasons of tact, 
you might frame the question more tentatively: “Do you think it would 
be advisable to talk to X—or do you think that would not be advisable?” 
Sometimes it is a good idea to ask the informant explicitly if his name can 
be used in seeking an appointment with the person he has suggested. 
This gives him an opportunity to protect himself if he does not want his 
name to be used and an opportunity to encourage name-dropping if he 
believes it will serve his interests. (That is, A may wish B to know that A 
has spoken of him as “a knowledgeable person,” or words to that effect.) 
Make sure that the informant provides sufficient contact information for 
you to locate anyone he recommends seeing. 

Knowing whom to stay away from is often an important by-product of 
inquiries such as these. If the informant is trusting and wishes to be help- 
ful, he may volunteer a cautionary aside such as, “If you do go to see X, 
youll probably find her reserved if not unsympathetic.” Unless X is a very 
important step in the developmental sequence at that moment, this may 
very well be a clue not to approach her until better groundwork has been 
laid for such a meeting. Another important by-product of such inquiries 
is a file of information on who is friendly, or antagonistic, to whom. Such 
information will be useful in constructing a map of political and admin- 
istrative feasibility for any new program that you may eventually propose. 


People leading to documents. Just as you can ask informants whom else 
to see or talk to, you can also ask them what else to read and how to 
obtain it. In visiting informants in their offices, you can sometimes get 
useful hints by scanning the bookshelves and the papers on tables and 
desktops for titles and authors or agency names. Also, take away from the 
interview all the documents that the informant is willing to give you, 
even if you are not sure how relevant they are. The chances are good that 
you will turn up some interesting new material in the collection you 
eventually develop, and, in any case, you may avoid a trip to the library 
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should you later wish to quote these documents or to report precise 
bibliographic information. Finally, put yourself on mailing lists, so as to 
be on the receiving end of whatever stream of reports, bulletins, newslet- 
ters, circulars, and so on are distributed by organizations operating in the 
policy area. Many agencies keep budgetary and other numerical informa- 
tion in electronic spreadsheet form; ask if the files can be sent to you or, 
better yet, copy the files to a USB flash drive before you leave the office. 


Documents leading to documents. Anyone who has ever written a sub- 
stantial academic research paper in history or the social sciences has 
probably learned how to use one document to discover another through 
Web links, footnotes, and bibliographies. The same procedures work in 
policy research. In addition, a researcher frequently uncovers references 
that are incomplete from a strictly academic point of view but that may 
still be useful for policy research. These include references to agencies or 
organizations (and even individuals) that have an ongoing responsibility 
for or interest in the policy area, some of whom can be expected to spon- 
sor studies, reports, position papers, and so on that may prove invaluable. 

Once research is under way, documents lead to documents in a rela- 
tively straightforward manner and without much difficulty. The problem 
is in knowing where to start when the research effort is just beginning. 
The easiest place to begin is the Internet, where Google or some other 
search engine can be used to find the sites of advocacy groups putting 
forth their views of the problem and possible solutions. These sites prob- 
ably contain valuable information and are a useful source of ideas and 
further leads. Because they are likely to be one-sided, however, you 
should try to find advocacy sites with opposing views. 

But advocacy groups are just a beginning. More useful are the Web sites 
of policy think tanks, such as the Brookings Institution and the American 
Enterprise Institute. These are relatively mainstream institutions that pro- 
duce large numbers of policy-relevant papers annually in almost all policy 
domains. The best of these papers connect concepts from the social sci- 
ences (often by noted scholars) with applied problems, and they often 
provide an overview of some policy area. Brookings is sometimes said to 
be a “liberal” think tank and AFI “conservative.” There is some truth in 
these characterizations, but such labeling is not as important as the fact 
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that each of these institutions cares about its reputation for sound ana- 
lytic work. It might be fair to say that they have political “orientations” 
rather than “biases.” The same is generally true of the “liberal” Center on 
Budget and Policy Priorities, the “conservative” Heritage Foundation, and 
the “libertarian” Cato Institute. In the environmental area, the leading 
think tank is Resources for the Future, which favors a benefit-cost 
approach to environmental policy. 

The Web sites of various governmental oversight institutions can be 
very helpful once you have in mind a particular legislative or regulatory 
issue. The Congressional Budget Office (www.cbo.gov) does hundreds of 
studies per year and posts many online. The Government Accountability 
Office (www.gao.gov) does the same. 

Do not be satisfied with only the sites that are accessible by means of a 
public-domain search engine. If you have access to the online resources of 
a university (or governmental) library, use it. University or government- 
wide libraries typically subscribe to databases that can provide access to 
full-text newspaper and magazine articles (Lexis-Nexis), as well as to 
abstracts and full-text publications in scholarly journals (JSTOR, in par- 
ticular). The CQ Researcher Online, published by CQ Press, provides 
access to feature-length journalistic articles dating back to 1991. National 
Journal is similar to The CQ Researcher. 

Because Internet sources are so accessible, it is easy to forget about 
books (until the day they are all online, of course). Unfortunately, elec- 
tronic search procedures do not work as well for finding good and appro- 
priate books as they do for finding articles and relatively ephemeral 
materials. The best way to locate relevant book-length sources is to find 
out what the experts and advocates recommend. You can check the bib- 
liographies in journal articles or— following the “people leading to docu- 
ments” strategy—ask them. 


Documents leading to people. Once having read, or read about, the work 
done by certain experts, academic or otherwise, you may wish to consult 
with them face-to-face or by telephone. You should be wary, however, of 
mistaking the nominal author of a study for the real one, particularly when 
that author is a person or group in officialdom. The nominal authors of 
Supreme Court decisions, to take an extreme example, are the Associate 
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Justices, but the real authors are usually their clerks, who in turn probably 
draw most of their arguments from the briefs filed by the lawyers on the 
case. Similarly, you should look behind the agency official whose name 
appears on the cover of a report, to locate the staff member(s) who did the 
work and may be named on the inside pages or referred to in a preface. 


Seeking Secondhand Information 


To find out what Senator A is doing or thinking about a policy problem, 
you need not necessarily ask the senator herself. Tens or hundreds of 
individuals may know the answer, or at least part of the answer. Such 
secondhand information must be used cautiously and checked constantly 
for bias or error. But it is not in any a priori sense inferior to information 
obtained firsthand, which may have its own biases and factual errors. To 
use a legal analogy, one relies for “truth” on witnesses rather than on the 
defendant, who, after all, cannot easily or prudently be asked to testify 
against himself. Sometimes it makes sense to obtain firsthand informa- 
tion as a supplement to the other, particularly if there is reason to think 
that failure to do so might ultimately jeopardize the credibility of the 
final research product. 

The use of secondhand sources is especially important in seeking 
political feasibility data. Suppose, for example, that you are planning to 
recommend that emergency ambulance services be centralized under the 
city police department, and you want to estimate the probable reaction 
of the fire chief to such a recommendation. You could ask the fire chief 
himself, but he might not be willing to tell the truth, especially if he were 
going to hold out his acquiescence in return for better terms or for some 
reciprocal benefit. That is, he might in principle be willing to go along 
with the change—he might even be enthusiastic about it—but for bar- 
gaining purposes he might not be prepared to say so. On the other hand, 
he might really be against it but not be willing to admit that, lest people 
call him an obstructionist. In either case, the fire chief is not a reliable 
source of this information. Eventually it might be desirable to ask him his 
opinion directly, but you could probably learn as much or more by ask- 
ing instead a variety of secondhand sources, such as a veteran city hall 
reporter, rank-and-file firefighters, someone in the city manager’s office, 
and someone from the police department. 
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Finding Multiple Sources of Firsthand Information 


Suppose that you wish to know about the past relationship between the 
police and fire departments. Have they been relatively cooperative, 
antagonistic, or indifferent? If for some reason you do not wish to ask the 
fire chief, it is always possible to ask the police chief, since she has also 
been a partner to these relations. Her view or interpretation of the rela- 
tionship may differ from that of the fire chief, but she is as much a par- 
ticipant and her knowledge just as direct. 

This principle has numerous applications. If you want to know what 
happened at a particular meeting to which you were denied admission (or 
to which you could not go for other reasons), there are many participants 
to query. If you want to know how one particular participant behaved at 
that meeting, you do not necessarily have to ask that participant. You can 
ask others who attended. If you wish to see a memorandum sent by Smith 
to Jones, you can ask either Smith or Jones, depending on which one you 
believe will be more agreeable—or you can obtain a photocopy from a 
third party.! 


Searching for Sources and Searching for Knowledge 


At the beginning of a policy research project, you face a dual uncertainty: 
about what you think you ought to know and about where you can turn 
to learn it. These are interdependent questions, in the sense that the 
reduction of one type of uncertainty is both a consequence of and a con- 
dition for the reduction of the other. 


1. The notion of systematically using secondhand sources and the notion of finding 
multiple sources for firsthand information are foreign to the spirit and practice of 
much social science research, which typically assumes that when you want to know 
the mental states or the conduct of a given individual, the best source is that indi- 
vidual. Such research then worries about how to devise measuring instruments and 
interviews that will register these facts about the individual with the least distortion. 
Often this is quite appropriate for the questions requiring basic and original 
research, when the object is to get pure data for pure understanding. But in policy 
research, the problem is to get a sufficient understanding of the world to be able to 
make estimates about alternative courses of action. Since there is much uncertainty 
about the future, and so many uncontrollable variables that will enter into future 
action, too much precision about the past and present frequently gets in the way. 
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Consider first what happens as you clarify your ideas about what 
you think you ought to know. Simultaneously, you are able to exclude 
certain sources you would otherwise have consulted and, because you 
know better what your objectives are, you are able to intensify your 
search for sources of greater relevance. This is the classic research 
model, in which ends determine means—that is, a constantly evolving 
set of knowledge objectives gives shape to the strategy of source selec- 
tion and consultation. It is as applicable to policy research as to any 
other sort of social inquiry. 

Its exact opposite is also applicable. Because the cost of searching for 
adequate sources is so high in time and energy, when you find a rich 
source it is wise to mine it intensively, even if that decision slightly alters 
your original knowledge objectives. If you wish to make recommenda- 
tions to the state legislature concerning the reduction of criminal recidi- 
vism rates, for instance, the most relevant data source (recidivism in that 
particular state) may not be as rich—and therefore as useful—as data 
from the bureau of criminal statistics run by some state that does an 
especially good job of collecting such data. 

One danger in this sort of pragmatism is that you may spend too much 
of your time on what appears to be a rich source, not knowing that there 
are much richer ones just around the corner. That is why it is wise to invest 
a good deal of time initially in canvassing a variety of possible sources and 
developing a broad overview of both the policy area and what means there 
are to learn about it. After this initial survey, it is possible to return to 
sources that look unusually rich. This procedure also guards against the 
second, and more important, danger in letting the sources guide you: 
you might lose sight of more desirable and feasible knowledge objectives. 
In the final analysis, there must be a balance between the classic model 
of ends (knowledge) dictating means (sources) and the pragmatic model of 
ends evolving out of the means one has at hand. 


GAINING ACCESS AND ENGAGING ASSISTANCE 


Gaining access can be a problem. If you wish to interview Assembly 
Member Jones, you must persuade Jones’s appointments secretary that 
you are on serious business and that in any event you will not be put off. 
You must arrange an appointment for a not-too-distant date and persist 
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even after Jones breaks the first appointment and fails to show up for the 
one made in lieu of that one. If you may wish to interview Jones a second 
time, you must take pains to keep this possibility open, and perhaps to 
foster it by your conduct during the first interview. 

You may need to engage the active assistance of some informants, 
especially those who stand at the gateway to an agency’s performance 
and budgetary data. Often these data are in a raw state—that is, the data 
are in the files but need to be collated and tabulated. Sometimes the data 
are in a semiprocessed condition; that is, they have been collated and 
tabulated, but they have not been put in a format intelligible to the 
researcher. (They are still in a format that is intelligible to the program 
managers, but this format does not fully reveal the meaning of the data 
to the researcher.) In such a case, you may wish to know about seeming 
inconsistencies in the classification of cases, or about the meaning of 
certain class designations that the managers have developed for their 
own decision making. 

Finally, there are data that have been prepared for public use but have 
not been processed completely or adequately for your purposes. Suppose 
that the intramural evaluation staff of a state penal institution, for exam- 
ple, has issued its annual report on releases and recidivists, but you can- 
not tell from the report how reliably they have ascertained the prior 
arrest and conviction records of the so-called first offenders. Did they 
rely on probation officer reports? On prison records? Records from other 
states? The error structure of an agency’s data is often not known to the 
agency, and if it is, it may not be made known to the public. In this case, 
as in the case of raw and semiprocessed data, interpretive assistance is 
needed from the agency itself. How much assistance it is willing to give 
may depend, in part at least, on how well you have established rapport 
with the agency and its personnel. 


Getting an Appointment 


Why should any informant grant you, a mere policy researcher, an inter- 
view? American manners and mores provide the most compelling reason— 
it is part of our definition of courtesy. If someone talks to you, even 
through your appointments secretary, you are supposed to talk back. Of 
course, the more powerful, busy, or politically defensive the personage 


ASSEMBLING EVIDENCE 91 


besought, the less will be the force of simple courtesy. In such cases, you 
might try to appeal to a sense of noblesse oblige or, if you have a presti- 
gious institutional affiliation, to a willingness to exercise your caste privi- 
leges. In addition, many people simply feel flattered by the interest of an 
outsider—even a policy researcher—who wants to listen to them. 

More reliable than these appeals to courtesy or vanity, however, is an 
appeal to political self-interest. Try to indicate that the outcome of your 
research is likely to have a bearing on the interviewee’s (or her agency’s) 
political fortunes and ambitions. It would therefore be prudent for her to 
be cooperative, to arrange for you to hear her (or her agency’s) point of 
view, and indeed to use the interview setting to assess the relevant politi- 
cal implications of your work. Of course, it may require some fast talking 
over the telephone, when you call for an appointment, in order to set her 
mind thinking in these directions. In dealing with an appointments sec- 
retary, who will probably be even less sensitive to your political cues, you 
may have to make your points indelicately explicit. Instead of relying on 
the vagaries of a telephone conversation or an appointments secretary, it 
may be useful to write a letter requesting an interview, followed up by a 
telephone call. 

Your informants will often be acquainted with one another and will 
occasionally talk among themselves about you and your work. Since you 
want such discussions to serve your interests rather than to work against 
them, you should try to develop a reputation as a competent, knowl- 
edgeable, and energetic researcher who is likely to produce something of 
intellectual or political significance. The best way to develop such a 
reputation is actually to be such a person, but, in addition, certain 
stratagems may prove useful. Attempt, for instance, to become a familiar 
face, by attending meetings and conferences that your potential infor- 
mants attend, and by loitering around office cafeterias or after-hours 
places that they frequent. Try to impress people with your ability to gain 
entrée to meetings that are only quasi-public in nature, and by talking 
in public places to important personages. All this familiarity will back- 
fire if you appear pesky or inept, so some judiciousness is in order. Also, 
you should appear to be learning quickly and critically while in these 
settings, rather than observing passively and dully. A notebook or laptop 
computer, in which you enter notes fast and furiously, is a good stage 
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prop as well as useful in its own right. Likewise, animated conversation, 
preferably observed rather than overheard, can enhance your appear- 
ance in these settings. But do not be indiscreet by becoming a bearer of 
information from one interviewee to another. 

Fieldwork does not proceed rapidly or smoothly. For the most part, 
you are a hostage to other people’s schedules. You can expect delays of 
several days to several weeks between the time you request an appoint- 
ment and the appointment date—and even longer if your informant 
eventually breaks the appointment and reschedules it for a few weeks 
later. (Sometimes it seems that research is mainly idle waiting!) This 
problem is particularly acute if delay in seeing one informant becomes 
a bottleneck to seeing others. To minimize idleness, it is a good idea to 
have two or three independent streams of interviewing running simul- 
taneously, so that a bottleneck in any single stream cannot halt your 
work altogether. 


Cultivating Access 


Securing repeated access to an individual or agency presents different 
problems from securing a one-time-only appointment. Courtesy is of 
almost no use here; the political motive, conversely, is critical. Since the 
political impact of your work on certain individuals and organizations 
will almost certainly be adverse, some doors will inevitably be closed to 
you. Beyond a certain point, there is nothing to be done about them, 
except to seek alternative means of entry. A perceived political affinity 
helps, but not much. Repeated access depends, instead, on building per- 
sonal rapport. This takes time, especially if you are not inclined to appear 
more friendly and congenial than you really feel. Rapport follows most 
of all from simple exposure. Think of yourself as an anthropologist who 
has to spend several months living among the tribe you are studying 
before being allowed to observe certain sacred rituals and practices. 

At the risk of sounding patronizing, I will nevertheless note here that 
the researcher should observe the basic courtesies. Be on time. Dress 
appropriately, which generally means with the same degree of formality 
as the interviewee or just a little less. Be friendly without being overly 
familiar or presumptuous. If you tape interviews—always a good idea, in 
order to preserve a record—set up your equipment with minimal fuss 
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and explain that the tapes are for your own reference only. State that you 
will turn off the tape whenever the informant wishes you to do so.” 

Almost invariably, whoever actually assists you in collating and inter- 
preting agency data will see himself as “doing you a favor,” regardless of 
how insistent his superiors have been that he make his services freely and 
generously available to you. As part of the protocol for such a “favor,” you 
must reciprocate with expressions of gratitude for his “going out of his 
way.” An even more cooperative informant might mail you a copy of a 
speech she has recently given, knowing that it will be of interest to you. 
Or she might see to it that you are put on the list of invitees to a banquet 
at which you will be able to meet a number of potential informants in an 
informal setting. To a certain extent, this sort of assistance can be encour- 
aged simply by letting people know that it will be welcomed. It can be 
facilitated by offering telephone and fax numbers or e-mail and postal 
addresses where you can be reached or where messages can be left for 
you. It may even be useful to have business cards printed with this infor- 
mation; relative to other research expenses, this one is quite small and 
can return high dividends. 


Exhausting Access 


Access can be exhausted, too, not just cultivated and built up. Whereas in 
some cases repeated exposure helps the researcher to build rapport, in 
others exposure simply tears it down. In the extreme instance, one expo- 
sure is all the relationship will bear; this commonly occurs when the 
informant is defensive or antagonistic, or when she is extremely busy and 
cannot easily be imposed upon. Other instances are intermediate: the 
informant is willing to grant two but not three interviews—or three but 
not four. When you suspect that access to an informant may be exhaust- 
ed relatively quickly, defer interviewing her until later in the research 
process, principally because your accumulated knowledge will then sup- 
port a more productive interview. 

Usually, deferring interviews with such informants inflicts no hardship 
on the researcher, since in the earliest stages, research can be conducted by 


2. If you come to sensitive material in the interview, remind the interviewee of your 
earlier offer to turn off the tape recorder. 
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talking with the legion of lower-level officials and administrative assis- 
tants, public relations officers, and so on. Potentially useful information 
sources are to be found among retired officials and among agency officials 
who are part of a dissident faction.’ These are rich sources at any time, but 
they are especially valuable in the early stages of research when it seems 
advantageous to defer your approach to more highly placed figures in the 
political establishment. 

The researcher’s reputation is also susceptible to being exhausted. It 
is perhaps not in danger of being lost, strictly speaking, so much as it is 
vulnerable to being transformed into a liability. Instead of being thought 
of as fair-minded, discreet, intelligent, and self-possessed, you may begin 
to be regarded as a partisan, a tale-bearer, a dope, or a dupe. The best way 
to avoid acquiring such an undesirable reputation is to eschew partisan- 
ship and indiscretion and, as I have already indicated, to actually be intel- 
ligent and self-possessed. 


CONDUCTING A POLICY RESEARCH INTERVIEW 


Policy research, in its completed form, becomes a political resource. 
Whatever its merits or demerits as a piece of rational analysis, it 
amounts to more than that. It may become a justification for certain 
parties to attack others or to defend themselves against attack and, 
hence, can be a weapon of persuasion in a war of propaganda. Although 
the tone and format of published policy research are typically neutral 
and disinterested, everyone recognizes that the research may be and 
often is used for political purposes, either by the author or by others. As 
a result, informants are highly sensitive to the political implications of 
whatever they tell you. How an informant treats you depends in large 
part on how she thinks your work will be brought to bear on her per- 
sonal or political interests. 

Being wary of the possible political implications of what they might 
reveal, informants may be reluctant to talk freely and honestly. You should 
assume that all interviewees confront this problem, even though you may 


3. My former Goldman School colleague William Niskanen relates that colonels 
twice passed over for promotion to general were a favored source for civilian policy 
analysts like himself in the U.S. Defense Department. 
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not know to what degree. In more extreme cases, it may be necessary to 
use various subtle kinds of leverage against the interviewee. Before turn- 
ing to the problem in its most severe forms, though, I will sketch a basic 
strategy for conducting policy research interviews in general. 


Energizing and Steering the Conversation 


The interview process is an interaction carried on between the informant 
and yourself. In this process the principal source of energy should be the 
informant. Your tasks are, first, to encourage the informant to talk and to 
keep on talking, and, once a suitable momentum has been attained, to 
steer, to redirect, to slow down, or to cross-examine. 

In addition to the political motivation, informants will talk because 
they have a story to tell. It is safe to say that many politicians, administra- 
tors, and important staff feel (correctly) that much of their best and most 
valuable work, which is being done behind the scenes, is unnoticed and 
underappreciated. They will be surprisingly eager to use you as their 
conduit to the outside world. Some also want to make their “side of the 
story” better understood than they think it is—and, if you haven’t heard 
it from others yet, you may be surprised at how interesting it is. 

In most social science research involving interviews, it is assumed that 
the interviewer is, as much as possible, a neutral instrument for recording 
data emitted by the respondent. However, this is generally an inappropri- 
ate model for policy research interviews. Here the informant assumes 
that you as an interviewer are anything but a neutral instrument—and it 
would be foolish for you to try to appear in such an ill-fitting disguise— 
since the whole object of your research is to arrive at some policy recom- 
mendations. Thus, you need not fear probing the informant with pro- 
vocative and even argumentative questions or comments—or to answer 
questions in return. Such exchanges can cause an informant to sharpen 
her wits and tone up her memory, and they may raise her psychic metab- 
olism sufficiently to infuse energy into the whole interview process. If 
this is done with proper finesse, the informant will appreciate the stimu- 
lation. Your finesse as an interviewer, of course, consists of being argu- 
mentative without sounding (or being) closed-minded or hostile. It is a 
good idea to introduce contentious remarks in such a way that the infor- 
mant, should she wish to do so, can retreat gracefully from the matter at 
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hand into another topic—thus keeping her energy level up rather than 
dropping into an embarrassed reticence. 

Most interviews are conducted at the informant’s place of work. 
Sometimes, however, a more informal setting, such as a restaurant or 
café, should be chosen. Your method of note-taking should be compati- 
ble with such an informal setting—perhaps on the back of an envelope 
handily stored in your pocket for just such occasions. 

Apart from energizing the informant, your other main function in the 
interview process is to steer her onto topics of interest to you. How can 
this be done? 

Sometimes you must interfere in the informant’s conversation stream 
simply to reestablish your right to speak, temporarily slowing down the 
informant without making her lose too much momentum. This can be 
done by interrupting with a short string of easily answered factual ques- 
tions pertaining to the subject matter she has been discussing. The con- 
tent of these questions, or at least the last one in the string, should be 
suitable to work as a transition to the next topic you have in mind. 
Suppose, for example, that you are interviewing the integrated-social- 
services coordinator in your county, who is telling you about her agency’s 
relations with the county’s chief administrative officer (CAO). Having 
heard enough on this subject, you now want to steer her onto her agency’s 
current budget request to the U.S. Department of Housing and Urban 
Development (HUD). The conversation might go like this: 


INFORMANT: ...so you see we've had a devil of a fight with the 
CAO all the way. Maybe it’s not her fault, of course, 
the Board of Supervisors being so conservative and 
the CAO needing support for her reappointment... 

RESEARCHER: [Interrupting] Yes, she is up for reappointment this 
year, isn’t she? 

INFORMANT: Yes. 

RESEARCHER: Well at least she doesn’t control your budget, does she? 

INFORMANT: True enough. 

RESEARCHER: But HUD does—and how are your relations with 
them? Do you get pretty much what you ask for from 
them, in the way of a budget, I mean? 
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The point is not to disguise from the informant the fact that you are 
trying to steer her away from one topic and onto another, although 
sometimes this is desirable and should be attempted. The point is really 
to help her move from one topic to another without having to lose 
momentum or to feel awkward. Indeed, she will sometimes feel trapped 
on a topic that she herself would prefer to leave, and your job at such 
moments is to help her maneuver off the subject. If you cannot think of 
where you wish to lead her next, just think of a subject that is not implau- 
sible and that is not too demanding emotionally or intellectually. While 
you go in slow motion through that topic, both you and the informant 
will have a chance to collect your thoughts and feelings preparatory to 
moving to the next matter of serious concern. 

Involving an informant in discussions of personalities is a delicate 
matter. The informant must be reassured that you are not turning the 
interview into a gossip session, that she is not a purveyor of gossip, and 
that you are not a seeker of it. This can be done by first introducing the 
name of the personality in a neutral, usually factual, context: 


RESEARCHER: A few moments ago you mentioned the Southside 
Community Health League. Dr. Green has been head 
of that for about a year now—or is it two? 

INFORMANT: Probably closer to two. 

RESEARCHER: Maybe it just seems shorter because I remember 
Dr. Black, his predecessor, so vividly. 

INFORMANT: Yes, Black was quite a leader there. 

RESEARCHER: Seems people have been more critical of Green— 
though I have heard quite complimentary things 
from some sources. 

INFORMANT: Yes, he’s pretty controversial. He’s certainly a competent 
administrator and has been pretty nice to us—though 
we deal mainly with his deputy, Mr. White. 

RESEARCHER: How come? 


Thus the conversation is turned to personalities by a sequence of 
small steps, in which each participant encourages the other and in which 
both assume responsibility for whatever gossipy quality may eventually 
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threaten to intrude. Since personalities are such a sensitive topic, it is 
even a good idea to sprinkle your conversation with allusions to people 
about whom you may have no desire to question the informant. When 
you do want to pursue a discussion of a particular personality, this pro- 
cedure makes the discussion seem less of a departure from the normal 
course of topics. 

If the informant has unpleasant things to say about the personality 
under discussion, you may want to take pains to establish your own 
social, personal, and political distance from that individual. In the exam- 
ple just given, for example, the researcher has referred to “Dr. Green” 
rather than “Bill Green” and has indicated his distance by suggesting that 
he is unfamiliar with certain particulars of Green’s career. If the infor- 
mant has flattering things to say about the individual in question, you 
may choose to follow a contrary course, though it is always a little risky 
to appear very close to anybody, lest it arouse suspicions of partiality. 


Leveraging the Defensive Informant 


Occasionally you encounter an informant who is irrevocably committed 
to a defensive posture, for whom “No comment” is the primary safeguard 
and calculated evasion is the fallback position. Try to diagnose this prob- 
lem very early in the interview and then reassess your goals for the inter- 
view in light of it. Concentrate on gaining information about specific 
questions that this informant is able to answer but that are probably not 
answerable by any other source. Since so much of your energy will have 
to go into cracking the informant’s defenses, focus on some very specific 
objectives and begin to probe for them right away. 

Once these preliminary assessments are out of the way and the inter- 
view has turned to specifics, the use of leverage is in order. First, let the 
informant understand that you are aware of his defensive posture, and 
signal that you do not intend to be put off by it. You might try to com- 
municate that his defensiveness will not help him, that you know too 
much already to be shunted aside, and that you have access to other 
sources who have already told you much and to still others who will be 
willing to tell you more. Indicate that information from these sources 
may be more prejudicial to his interests than his own revelations would 
be, and that he therefore has nothing to lose, and perhaps something to 
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gain, by giving honest answers. A certain amount of bluffing may some- 


times be necessary, though this tactic carries obvious risks. It is always 


better to actually know as much as you pretend to know, and to have 


access to the sources you claim to have access to, than merely to bluff. 


Here is a sample of such an interview, with the head of a prominent 


local insurance company whom the researcher is pressing hard: 


RESEARCHER: 


INFORMANT: 


RESEARCHER: 


INFORMANT: 
RESEARCHER: 


INFORMANT: 
RESEARCHER: 


INFORMANT: 


RESEARCHER: 


INFORMANT: 


One thing I'd like to get more information about is 
the problem insurance companies have writing 
policies for merchants in so-called ghetto areas. 
[Silence. Pause. ] 

I mean, there may be problems because these policies 
are risky business propositions. 

[Silence. Pause. ] 

People say they are risky, anyway. Do underwriters in 
this area consider them risky? 

I can’t really say for sure. 

Well, some people in the Black Merchants Association 
claim that insurance companies won't write policies 
for them at all, that they’ve been classed as 
“unacceptable risks.” 

I don’t really know—insurance writing is the science 
of risks, isn’t it? 

[Decides that informant will provide no information 
on insurance industry doctrines or practices in general, 
or on the local underwriters in particular. Guesses that 
informant will be unwilling to discuss the doctrines or 
rules applied by his own company, and decides 
therefore to concentrate solely on gathering 
information about the practices of the informant’s 
company.] Perhaps I can clarify my question by being 
more concrete. In your own Bedrock Casualty 
Company, are applicants ever turned down because 
they are thought to be unacceptable risks? 

I can’t say for sure. I’m not that close to the operating 
details of our very large company. 
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RESEARCHER: 


INFORMANT: 


RESEARCHER: 


INFORMANT: 


RESEARCHER: 


Of course. [Signaling she will not be put off] You, or 
perhaps your secretary, could arrange for me to talk 
to someone at that level, though, couldnt you? 
[Seeking a different leverage point] But tell me about 
the category of “unacceptable risks.” Does Bedrock 
Casualty tell its salespeople that the company will 
insure any premises provided the insured pays a high 
enough premium? [Shifting the terms of the question 
to throw informant off guard] Or is there a limit on 
how high a premium the company will set? 

Well, we do not like to charge exorbitant premiums, 
of course ... 

[Interrupting] So within the existing limits on 
premiums there might in fact be businesses too risky 
to insure—hence “unacceptable”? [Holding to 
offensive] How about cancellations? Has Bedrock 
canceled or refused to renew any policies of ghetto 
merchants even though they have not filed any claims 
recently? This is another thing the Black Merchants 
Association has been complaining about. 

[Deciding researcher knows more than he had 
thought and seeking preemptive protection against 
the Black Merchants Association’s allegations] Well, 
yes, we have canceled a few, in the more riot-prone 
areas, and refused to renew other policies in that area. 
We had no choice; we stood to lose a lot of money in 
case of any trouble. 

[Graciously ignoring this “confession,” and trying to 
induce the informant to tell his side of the story] Of 
course, that’s quite understandable. I think most 
people recognize this problem. [Now taking aim on a 
single statistic, the proportion of all Bedrock policies 
in ghetto neighborhoods canceled or not renewed in 
the last two years] In the past, have you written many 
policies in that area? 


INFORMANT: 
RESEARCHER: 
INFORMANT: 


RESEARCHER: 


INFORMANT: 


RESEARCHER: 


INFORMANT: 


RESEARCHER: 


INFORMANT: 


RESEARCHER: 
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Yes, we’ve done quite a bit, in the past anyway. 

You still do insure some business over there, don’t you? 
Yes, we do, though as I say, I’m not too close to the 
operating details... 

[Interrupting] Could you estimate what proportion 
of your policy holders from, say, two years ago you 
continue to insure? Is it 80 percent, 20 percent? Just 
to give me some rough idea. 

Well, it would certainly be a lot closer to 80 than 20 
but I really don’t know. 

[Deciding that this would be an interesting datum and 
that it is worth pursuing vigorously] Can we find out? 
Not easily. It’s not in any files anywhere in that form, 
and it would be awfully difficult to find out. 

[Not believing that it would be very difficult, 
deciding to contribute her own labor to searching 
the files, if necessary, and resorting to a bluff] 
People have the impression that Bedrock is less 
inclined to write policies for ghetto merchants than 
other companies in this area. I don’t know where the 
facts come from—but I think some lawyers 
connected with the Black Merchants Association 
have been looking into legal aspects ... 

What? I’m sure we are no worse, or different, than 
any other company in town! Pd like to see these 
so-called facts! 

If I get any further clarification on that, Pd be happy 
to let you know. Meanwhile, Id be willing to help out 
in whatever way you like in getting this information 
together concerning your own company’s record in 


this field. 


Let us interrupt this scene without a conclusion because, however it 
turns out, the researcher has done the best she could. The president of 
Bedrock Casualty may deliver the information sought, or he may not. 
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Good interviewing strategy and tactics do not guarantee success, espe- 
cially when the odds are weighted against the researcher to begin with.* 

One common ploy used by a defensive informant is to reel off masses 
of irrelevant statistics and facts, which can easily swamp a naively data- 
worshipping researcher. Another ploy is to ramble garrulously about side 
issues, while running out the clock on whatever time limit has been set 
for the interview. Your best defense against these evasive tactics is to be 
able to recognize them for what they are. 

If your own leverage fails—and if the elusive information is sufficient- 
ly important to you—you may be able to use someone else’s. A graduate 
student researcher may have little leverage with determinedly defensive 
bureaucrats, for instance, but a legislator, or her staff assistant, will almost 
certainly have more. Hence, as a last resort, you might persuade a sympa- 
thetic legislator to help out. Sometimes a newspaper reporter or an estab- 
lished group can be of assistance. The local medical society, for example, 
may be able to get information from the county hospital administrator 
about hospital policies that no academic researcher—and perhaps not 
even a county supervisor—could get. 

A significant constraint on using leverage is the desirability of main- 
taining cordial relations with whatever agency or individual is being 
pressured, for you run a clear risk of alienating the objects of your lever- 
aging tactics. With respect to a given study, this problem can be miti- 
gated by postponing the more offensive tactics until relatively late, when 
the study is less vulnerable to being undermined by the offended party. 
The problem is more difficult, however, when you envision a long-term 
relationship—lasting well beyond the conclusion of the present research 


4. The researcher’s bluffing tactic in this scene is of debatable morality. Although I 
believe it would be unethical in most circumstances, there are occasions when it can 
be justified. This is one of them. In this case, the Bedrock president seeks to withhold 
proprietary information. Does he have a right to do so? Normally, yes. But this right 
has to be weighed against the injustice of depriving ghetto merchants of a nearly 
essential prerequisite for doing business when they might be perfectly willing to 
meet reasonable price terms for acquiring the insurance (perhaps with government 
or philanthropic assistance). The researcher here has an arguable right to try to com- 
bat this injustice. Given that right, does she also have the right to use deception? The 
use of explicit deception on the part of the researcher is balanced by the use of 
implicit (covert) deception on the part of the insurance company’s president. 
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effort—with the agency or individual under scrutiny. Certain informa- 
tion may have to be sacrificed in order to preserve a modicum of good- 
will for the future. 


USING LANGUAGE TO CHARACTERIZE AND CALIBRATE 


The basic medium of the interview is spoken language embedded in a 
conversational context. Such a medium, when used as a representational 
device, presents reliability and validity issues (in psychometric terms). 


Eeemantic Tip The simplest issue—to see, though not necessarily to 
resolve—involves the language of characterization. If an informant says, 
“Yes, this is a frustrating job,” you have to interpret both the nature and 
intensity of the word frustrating, and do so in a way that permits you to 
calibrate the result against some larger frame or benchmark. This can be 
done by asking a series of questions designed to do the calibrating. One 
shortcut is to start by offering up your own characterization and see how 
the informant reacts to it: “If I had this job, I would find it awfully frus- 
trating, I think.” This quickly establishes a benchmark of some kind— 
“awfully frustrating”’—for you and the informant to use. Of course, there 
is the problem of knowing whether you and your informant mean the 
same thing by the expression, since your frustration thresholds may dif- 
fer. But youre off to a good start. 

An improvement on the previous example would be to create two such 
benchmarks—that is, to describe a whole continuum with anchors at 
both ends and perhaps a verbal midpoint. For example: “Would you say 
that your reaction to proposal X was extremely skeptical—as I’ve inferred 
from what you already have said—or was it relatively favorable. ... or was 
it maybe ‘wait-and-see’?” This approach has the added advantage of 
respecting virtually any position your informant holds and of communi- 
cating your willingness to find anchoring words based in the informant’s 
own history. Or you could anchor one or both ends in what “other 
people” have supposedly been saying. 

To be sure, by characterizing the available options in this way, you are 
putting words into other people’s heads and sometimes into their 
mouths. Before you proposed “extremely skeptical,” your informant may 
never have thought of the proposal in this way, and so you run the risk, 
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by asking the question, of having created such a thought out of thin air. 
But that risk comes from using language as a medium; it can’t be avoided. 
Even when you use the ostensibly neutral and clinical language that sur- 
vey researchers and reporters use, you are putting words into people’s 
heads and mouths. More provocative characterizations, when used as 
benchmarks, are on a logical par with the more neutral alternatives 
offered by survey instruments and professional journalists. 


PROTECTING CREDIBILITY 


Like social science research, policy research is eventually subject to criti- 
cism on intellectual grounds. But unlike social science research, it is even 
more vulnerable on political grounds and, indeed, is vulnerable to attack 
by the very subjects of the study. In social science research, the subjects 
rarely become significant critics of the product, but in policy research 
their criticism is inevitable. Therefore, the researcher should take steps to 
protect the ultimate political credibility of his work from politically 
motivated as well as strictly intellectual attack. 


Defending against Politically Inspired Criticism 


In contrast to that of social science research, the primary goal of policy 
research is not intellectual enlightenment (either yours or that of your pro- 
fessional colleagues), although enlightenment is inevitably a by-product. 
Instead, the goal is to improve your understanding of a policy problem, 
and of possible means of coping with it, to the point at which it becomes 
possible to advocate a responsible course of action. Thus, policy research 
takes aim at broad and complex phenomena, and so it is typically satisfied 
with very gross approximations of “truth,” in contrast to social science 
research, which typically seeks more refined interpretations of narrowly 
circumscribed problems. The gross and approximate character of policy 
research is an open invitation to politically inspired criticism. How can 
you, as the researcher, protect yourself? 

For one thing, you should attempt to touch base with any party (or 
any institutional interest) who might later try to undermine the report by 
claiming to have been ignored. Indeed, it is a good idea to preempt such 
claims by quoting that party in the report, as evidence of a sort that the 
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party’s views were taken into account. For instance, if you are going to 
recommend alterations in the way superintendents are selected in a given 
school district, it would be best to interview representatives from the 
local association of school administrators and from the local chapters of 
the National Education Association and the American Federation of 
Teachers. Spokespersons for these groups may have interesting opinions 
to contribute to the research project, but even if they do not, by consult- 
ing them you gain protection against their criticisms should they decide 
to oppose the recommendations in your report. It may even be useful to 
send out a preliminary copy of the report to these interests for reviews. 

Second, you should seek out “experts” or others with political or intel- 
lectual authority to whom you can attribute views, opinions, estimates, 
and so on, about which you feel especially uncertain. Quoting published 
sources is one way of making such attributions, and including quotations 
from interviews is another. In addition, you should line up experts who 
will be willing to speak up in support of your work once it becomes pub- 
lic. Sources who are quoted in the report as having a view on this or that 
subject become natural targets for inquiring journalists or political deci- 
sion makers; these sources have an incentive to defend their quoted views 
when questioned. 

Third, you should pay special attention to potential opponents and 
identify which propositions they are likely to attack. These target points 
should be bolstered in advance by expert quotations, and some polite 
reference should be made to the existence of counterarguments—with- 
out giving them too much space or prominence. The very opponents 
who can be expected to raise objections later should be quoted, to defuse 
any claims that their arguments or positions were ignored. (There may 
be additional psychological advantages to the balanced or two-sided pre- 
sentation, simply as a subtly persuasive form of propaganda directed at 
the reader.) 

Statistics can be useful for buttressing credibility. Employed for this 
purpose, they play a documentary rather than an informational role. 
Statistics can document the validity of generalizations that political oppo- 
nents might otherwise challenge, even though their truth is abundantly 
evident through more impressionistic sources. 
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Preparing for Premature Exposure 


Politicians and policy researchers work on different timetables. The for- 
mer often call for “results” well before the research is in any sense finished. 
Even when no one demands it, however, unexpected opportunities often 
do present themselves before your research work is close enough to being 
finished that you can seize the auspicious moment to present your results. 

One possible strategy is to map out (as much as possible) the timeta- 
ble of potential political demands and to arrange your research timetable 
in at least partial correspondence. Another strategy is to prepare yourself 
as soon as possible with answers to the crudest kinds of questions that 
might be asked of you. Since these are generally the kinds of answers 
politicians need and want anyway, you may as well formulate them early 
in the course of your research. Finally, it is important, early on, to line up 
your supporting experts, as well as to touch base with potential oppo- 
nents. Since, once again, these contacts must be made eventually, there is 
good reason to make them sooner rather than later. 


STRATEGIC DILEMMAS OF POLICY RESEARCH 


By way of summary and conclusion, let us consider the question: Which 
informants should be approached when? Answering this question forces 
a useful review of most of the issues discussed earlier. 

We may divide the “when” part of the question into “relatively early” 
and “relatively late” in the course of the research project. Approach the 
following informants relatively early: 


e Persons who are likely to facilitate your search for rich information 
sources 

e Powerful persons who directly or by your reputed connection with 
them will facilitate your access to sources 

e Knowledgeable persons who will provide you with the information 
you need to hedge against premature political exposure of your 
work, and whose information will contribute to your capacity to 
exert leverage against defensive interviewees 

e Friendly experts who will contribute to your political credibility in 
case of premature political exposure 
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e Potential opponents with whom you touch base in order to hedge 
against premature political exposure 


Approach these informants relatively late: 


e Hostile or defensive informants against whose tactics a prior build- 
up of leverage is desirable 

e Busy informants to whom you might lose access permanently once 
you have seen them, or about whom you are not sufficiently informed 
to interview early 

e Potential opponents, especially if powerful, who might try to fore- 
stall your access to others and thereby cripple your research efforts 

e Administrators who have knowledge of potential trouble spots but 
who will be unwilling to point them out until it appears to be in 
their self-interest 


There is one obvious contradiction between these two lists—approach 
potential opponents early and late—and several others that are not quite 
so obvious. Often the busy and the defensive informants are also in the 
best position to facilitate the search for sources, open doors, and provide 
useful information. Top agency administrators, for instance, may have 
plentiful experience with the policy problem under investigation and 
may be able to provide easy access to sources, but they also have a vested 
interest in maintaining the status quo or something very close to it. In 
any event, they may not take kindly to having their activities scrutinized 
too carefully by an outsider. Other similar examples can easily be called 
to mind. There is in principle no way to reconcile these incompatible 
prescriptions of whom to approach early and whom late. You will have to 
consider the full details of your particular situation and then balance the 
risks and rewards inherent in any given choice. There is no way of avoid- 
ing such trade-offs; you should simply make them consciously rather 
than inadvertently. 


PART 


I] 


“SMART (BEST) PRACTICES” RESEARCH: 
UNDERSTANDING AND MAKING USE 
OF WHAT LOOK LIKE GOOD IDEAS 
FROM SOMEWHERE ELSE 


t is only sensible to see what kinds of solutions have been tried in other 

jurisdictions, agencies, or locales. You want to look for those that appear 
to have worked pretty well, try to understand exactly how and why they 
may have worked, and evaluate their applicability to your own situation. ! 
In many circles this process is known as “best practices research.” Simple 
and commonsensical as this process sounds, it presents many method- 
ological and practical pitfalls. The most important of these is relying on 
anecdotes and on very limited empirical observations for your ideas. To 
some extent, these are—one hopes—supplemented by smart theorizing. 
This method is never perfectly satisfactory, but in the real world the alter- 
native is not usually more empiricism but, rather, no or thoughtless theo- 
rizing. Part III helps you to avoid the pitfalls and offers tips on how to get 
the most payoff from your search for best practices. 


DEVELOP REALISTIC EXPECTATIONS 


Semantic Tip| First, don’t be misled by the word best in so-called best 
practices research. Rarely will you have any confidence that some helpful- 
looking practice is actually the best among all those that address the same 


1. Readers interested in a more social scientific exposition of many of the points in 
Part HI should consult Bardach 2004. 
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problem or opportunity. The extensive and careful research needed to 
document a claim of “best” will almost never have been done. Usually, you 
will be looking for what, more modestly, might be called “good practices.” 

But even this claim may be too grand. Often you can’t be sure that what 
appears to be a good practice is even solving or ameliorating the problem 
to which it is nominally addressed. On closer inspection, a supposedly 
good practice may not be solving the problem at all. Inadequate measure- 
ment, plus someone’s rose-colored glasses, was simply producing the illu- 
sion of problem mitigation. It may also turn out that, even if good effects 
have truly occurred, the allegedly good practice had little or nothing to do 
with producing them. All these are known technically as internal validity 
problems. The discussion that follows assumes such problems have been 
satisfactorily—though not perfectly—resolved and concentrates only on 
the external validity problem of extrapolating from a setting in which a 
good practice has indeed worked well to settings that differ in little- 
understood but important ways may lead to weak, perverse, or otherwise 
damaging results. 


ANALYZE SMART PRACTICES 


A “practice” is a tangible and visible behavior. When you can ask a pro- 
gram manager what her practice is in addressing some problem, she can 
answer with a description of what she does. Typically, though, a practice 
is also an expression of some underlying idea—an idea about how the 
actions entailed by the practice work to solve a problem or achieve a goal. 
Some such ideas are particularly clever, and I shall explain further what I 
mean by this. The practices that embody them I call “smart practices.” 


Finding the Free Lunches 


One way of being clever is by getting something for nothing. Contrary to 
the dictum that there is no such thing as a free lunch, creative policymak- 
ers and policy implementers invest quite a lot of energy in looking for 
just such comestibles. Often, they are successful. To understand how this 
can be, consider the free lunch cornucopia produced by the natural sci- 
ences and engineering. The energy stored in the chemical bonds in a 
cup of gasoline can run a car for a few miles if only you know how to 
access that energy and channel it. Pulleys and levers supply mechanical 
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advantage. Bacteria happily eat and destroy the organic crud in a city’s 
wastewater almost for free. All these materials, devices, and conditions 
amount to getting a lot of “something” for nothing or for relatively little. 
The source of all these boons is simply Mother Nature. 

In the social world, the sources of something for nothing are usually 
less tangible and less directly gifts of Mother Nature, but they are no less 
real. The “invisible hand” of the market creates social value where once 
there was only individual pursuit of self-interest, and, metaphorically at 
least, it operates without charge. Alphabetical ordering permits people to 
find information in a fraction of the time it would have taken had there 
been no such ordering. Queuing at bus stops is easy to understand and 
usually fair, and it makes life better for everybody. 

In the world of policy and management, there are no doubt fewer and 
less delectable free (or nearly free) lunches than in the marketplace or in 
an information storage facility or at a bus stop. But they are there. All the 
“opportunities” described in Box I-1 (p. 8) have this latent potential to 
generate something of public value relatively cheaply. (On the nature of 
“public value,” see Moore 1996.) You might say that the difference 
between the (high) value created and the (low) cost, and risk, of produc- 
ing it represents a free lunch.” 

Opportunities don’t deliver up their latent value without some addi- 
tional work, however. This work is done by practices that take advantage 
of their potentialities, and these practices typically cost something and 
are subject to various vulnerabilities, as well. However, the smarter these 
practices are, the more value they can manage to extract at lower cost 
and risk. 

The following list offers some examples of candidates for smart practice 
status—candidates, that is, because to my knowledge they have not been 
subjected to the extensive empirical testing needed to confirm such status: 


e A “high-expectations” welfare-to-work program. Implemented in the 
early 1990s, the Greater Avenues to Independence (GAIN) program 
I studied in Riverside County, California, was a prototype for the 


2. Risks come in several varieties; see the section in Part III, below, titled “Describe 
Generic Vulnerabilities.” 
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1996 federal welfare reform act. Unlike most other welfare-to-work 
programs, the Riverside program set high expectations about work 
for GAIN participants in two senses. In many different ways, it sig- 
naled to participants that program staff had confidence in (high 
expectations of) their ultimate success in getting a job and getting 
off welfare. This confidence was intended as an antidote to many 
participants’ low self-esteem, and consequent low effort to reattach 
themselves to the labor force. Staff also signaled—and expressed in 
program rules about early and diligent job search, as well as through 
a variety of formal and informal pressures—that “society expected” 
welfare participants to shape up and take responsibility for their 
own financial well-being. The Riverside GAIN program designed its 
recruitment, training, performance appraisal, and other administra- 
tive systems to support this high-expectations philosophy (Bardach 
1997). In effect, the high-expectations model took advantage of the 
natural energy to solve their own problems that program managers 
assumed to be latent in the program participants. 

Reading One-to-One. This tutoring program for children in grades 
1-3 who have fallen far behind in learning to read English was cre- 
ated by George Farkas, at the time, of the social sciences faculty of the 
University of Texas at Arlington. It was first tried out in Dallas and 
then spread to Houston and a number of other cities. The program 
involves systematic instruction in phonemic awareness, one-to-one 
tutoring by a well-trained tutor, and highly structured feedback and 
supervision. Like all phonemics-based programs, it recognizes that 
English orthography does not map sounds in a systematic or logical 
way and that it is at some point necessary for learners to master the 
decoding and encoding rules actually in use. It takes advantage of the 
fact that children’s early failures in reading that come from neglect of 
phonemic awareness are reversible by regular tutoring. It also takes 
advantage of the emotional bonding that comes from the one-to-one 
tutoring relationship and of the increased motivation after the stu- 
dent experiences some successes. The simplicity and systematization 
of the teaching materials, teaching methods, and administrative over- 
sight system make the program easily replicable and keep the costs 
relatively low (Farkas 1998). 
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e Sharing maintenance responsibilities for a neighborhood park between 
the local parks department and the residents of the neighborhood. 
Nonprofit organizations often spring up to provide services of a 
nonstandard sort not provided by the public agency (e.g., same-sex 
schools, abortion clinics). An extension of this basic idea is a part- 
nership in which the public sector supplies certain resources that 
are not only supplementary but also complementary. In many a 
setting, the city government provides the parkland and the neigh- 
bors provide some or all of the labor to make the land more service- 
able in some way. This practice takes advantage of two interesting 
potentialities: the potential for gains from trade between two par- 
ties, and the use of what is in effect bartering, in a situation where 
there are administrative and political barriers to organizing the 
transaction in cash. 

e The “expenditure control” budget. Adopted first in the city of Fairfield, 
California, this practice was publicized by David Osborne and Ted 
Gaebler (1992) in their influential book, Reinventing Government. As 
originally conceived and implemented, this budgeting strategy gave 
each department the same basic mission and the same budget as in 
the previous year (with an inflation adjustment) but abolished the 
line-item specification of expenditures, permitting the department to 
keep any savings and reinvest them in other mission-related activi- 
ties. This approach took advantage of the superior technical and 
operational knowledge of program implementers relative to that of 
elected officials and bureaucrats in fiscal-control agencies. 

e Milestone payments to nonprofit service contractors. In 1992 the 
Oklahoma Department of Rehabilitation Services began paying non- 
profit contractors for meeting rehabilitation milestones, defined in 
performance terms, that mental health clients could achieve en route 
to higher levels of employability. The clients participated in assessing 
whether the milestones had been met, and the contractors helped 
to define generic milestones and other aspects of the program. 


3. This program was a 1997 finalist in the Ford Foundation/Kennedy School of 
Government (KSG) Innovations in American Government competition. My source 
for information about it was the Innovations program files. 
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The milestone system also permitted contractors to claim reimburse- 
ment from the state on a more accelerated schedule than they had 
previously been able to do, thereby taking advantage of the power of 
self-interest to motivate better performance from the nonprofits. It 
also provided greater transparency than more traditional fee-for- 
service arrangements, under which the funding agency did not know 
much about the quality of the service provided. 

e A cooperative project between rehabilitation and recycling programs. 
Hennepin County, Minnesota, arranged for mentally disabled cli- 
ents of the county vocational rehabilitation program to sort and 
recycle discarded auto batteries, an item of concern to the county’s 
environmental management agency.4 The two programs thus took 
advantage of production complementarities between human and 
physical “assets” that they could deploy. 


E] | have made a point, in describing supposedly smart prac- 
tices, of saying that each practice “takes advantage” of something. This is 
a linguistic device for ensuring that, in analyzing how the practice works, 
we focus on how the practice aims to exploit, or take advantage of, some 
latent opportunity for creating value on the cheap. 


Breaking Loose from Conventions and Assumptions 


Another way of being clever is not so much technical—finding those free 
lunches—as ideological and psychological. It involves disrespecting con- 
ventional boundaries. In the world of public policy and management, 
this practice sometimes involves challenging assumptions that are 
anchored in value commitments. For example, since the late 1980s we 
have begun to shake loose the assumption that just because some good 
or service is “good for the community” and ought to be financed 
through taxation, it ought also to be produced or delivered by government 
employees. Instead, we now contemplate contracting out to the nonprofit 
or even the profit-seeking sectors such traditionally “governmental” func- 
tions as primary education, correctional institution construction and 


4. This project was a Ford/KSG semifinalist. See Borins 1998, 200; also, I had per- 
sonal communication with Hennepin County program managers. 
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management, and welfare-to-work programming. In this case, we are 
challenging the assumption that governmental provision necessarily 
embodies a social expression of the value of community. Taxpayer financ- 
ing may do so, but governmental provision does not. 

Another value-oriented smart practice may be to simply articulate the 
values that underlie a program and make it effective. Riverside’s high- 
expectations welfare-to-work model, for instance, was one of relatively 
few programs prior to the mid-1990s that decided it was proper to articu- 
late the value premises that underlay its approach to case management. 


OBSERVE THE PRACTICE 


In free-lunch-type situations, we can say that the smart practice is “what- 
ever takes advantage of—or exploits—the latent opportunity to create 
value on the cheap.”® But let us try to say more about how to characterize 
this “whatever.” 


Characterizing the Features of a Smart Practice 


The basic mechanism in a smart practice is its means of directly accom- 
plishing useful work in a cost-effective manner. A smart practice is made 
up of (1) the latent potential for creating value (from Box I-1, for instance), 
plus (2) the mechanism for extracting and focusing that potential. In the 
six examples described earlier, I indicated the basic mechanisms by saying 
what each of the practices “takes advantage of.” For instance, the shared 
maintenance for parks takes advantage of potential gains from trade and 
the opportunity to use barter as a substitute for cash payment. 

But there is more to a smart practice than this basic mechanism 
(Bardach 2004). Some characteristic secondary features of a smart prac- 
tice are the following: 


e Implementing features, which directly embody the basic mechanism. 
In the Oklahoma milestones case, for instance, they are the pay- 
ment schedule, payment amounts, and payment conditions. In the 


5. Whether or not contracting-out is a smart practice, it is highly controversial, 
I might add. 

6. With minor adjustments, the same analysis can also be applied to practices whose 
“smartness” derives from their departure from convention. 
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Hennepin County recycling/rehabilitation program they are the 
stock of recyclable materials, the pool of mentally disabled clients, 
and the interagency understandings that link them. 

e Supportive features, which are primarily those resources used to 
bring the implementing features into being—for instance, a budget 
and an institutional structure. Other supportive features that have a 
less directly instrumental role but may nevertheless be important 
might include the culture of the organization or the broader politi- 
cal environment. 

e Optional features, or those that just happen to be of interest to actors 
in the site where the practice is observed but may not necessarily be 
valued elsewhere. For instance, in the Oklahoma milestones case, the 
feature that allows vendors to participate in the design of the pro- 
gram seems to me optional—although nice! 


Distinguishing Functions and Features 


Semantic Tip f adapting a seemingly smart practice from a “source site” 
for application at a “target site,” you want to be rigorous in replicating the 
logic—the “how?”—of the basic mechanism, while leaving maximum 
flexibility as to the specific means to carry it out. To do this, distinguish 
between the functions involved in getting the mechanism to work and the 
particular features that embody those functions. For instance, in the 
milestones program, the functions include setting the milestones and 
verifying the claims of achievement. These actions are part of the defin- 
ing logic of the practice—they cannot be omitted without changing the 
very essence of the program. However, exactly what features are chosen 
to implement these functions or to support the implementation strategy 
is another matter. With regard to the high-expectations welfare-to-work 
program, two essential functions are creating a moral climate: favoring 
responsibility and instilling self-confidence that such responsibility can 
be met. Exactly what design features should be chosen to implement and 
support these functions is a more open-ended question, though. 


E] Here is a linguistic hint to help you separate features and 
functions: Functions should be formulated as gerunds, verb-like nouns 
ending in “ing’—as in the actions defined above as setting, verifying, 
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creating, and instilling—while the features that perform these functions 
can be indicated by pure nouns. 

An exception to this principle of formulating functional language 
arises when you really need or want to specify a particular method for 
carrying out a function. In the milestones case, for instance, you might 
intentionally refer to a contract as a specific means of defining expecta- 
tions among the parties and to documents as a means of attesting that 
the milestones have been met. 


Allowing for Variation and Complexity 


Because smart practices are internally complex, context-sensitive, and 
capable of being used by different parties to pursue slightly different bun- 
dles of goals, how we talk about them should reflect these qualities. 


Characterization should be generic and flexible, not prescriptive and 
overly precise. Consider the expenditure control budget described ear- 
lier. Does the practice there require giving all the savings back to the 
department, or would, say, 50 percent qualify? If the basic idea is to pro- 
vide incentives to spend wisely, returning 50 percent may suffice. 
Probably the best characterization, therefore, would be “allowing the 
department to retain enough of the savings for its staff to feel motivated 
to create the savings in the first place.”” It would then be up to whoever 
implements the expenditure control budget to determine what “enough” 
means in the local context. 

It should be left to local implementers to figure out the details of the 
generic practice that make sense in their own context. Allowing for local 
adaptation of nonessential features not only serves common sense but also 
encourages greater buy-in by the locals to a practice that in some sense is 
being imported from elsewhere or, worse yet, imposed from outside. 


Characterization of the basic mechanism of a smart practice is not nec- 
essarily simple; it can be complex. My list of examples of candidate 
smart practices included only relatively simple practices, so as not to cause 


7. This interpretation is asserted by the researcher-observer; it is not necessarily 
something that has been done in practice or endorsed by any practitioners. 
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confusion. However, some smart practices are multifaceted, and thus not 
easy to summarize in a few sentences or even paragraphs. Michael 
Barzelay analyzed what he called the “postbureaucratic paradigm” for 
managing statewide overhead and control functions in Minnesota state 
government. He considered trying to reduce the many aspects of this 
postbureaucratic paradigm—which I would also call a smart practice, 
albeit a very large one—to a few “core ideas” such as service, customer 
focus, quality, incentives, creating value, and empowerment. However, he 
concluded, “the major concepts . . . are not organized hierarchically, with 
one master idea at the top,” but are instead arrayed as “an extended fam- 
ily of ideas” (Barzelay 1992, 115-117). 

A related management reform paradigm, called by many the “new 
public management” (NPM), emerged in New Zealand in the mid-1980s 
as another such extended family of ideas and practices. Noting that it “is 
not reducible to a few sentences, let alone a slogan,” one observer (Borins 
1998, 9) goes on to state its key ideas, as follows: 


e Government should provide high-quality services that citizens value. 

e The autonomy of public managers, particularly from central agency 
controls, should be increased. 

e Organizations and individuals should be evaluated and rewarded 
on the basis of how well they meet demanding performance targets. 

e Managers must be assured that the human and technological 
resources they need to perform well will be available to them. 

e Public-sector managers must appreciate the value of competition 
and maintain an open-minded attitude about which services belong 
in the private, rather than the public, sector. 


Specimens of a smart practice in the real world look rather different from 
one another and require careful interpretation. You should try to find 
multiple exemplars, or specimens, of a smart practice to get a sense of its 
robustness and efficacy when (1) it is being implemented under different 
supportive (or antagonistic) conditions, (2) it comes with different option- 
al features attached, and (3) it employs supposedly equivalent but never- 
theless somewhat different means to perform the required functions. 
Ideally, you would be able to find social scientific evaluation studies of 
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practices that supply both data and theoretical interpretation regarding 
such matters. In most cases, however, such evaluations will not exist. 
Normally—or perhaps at best—you will find writings or speeches by prac- 
titioners describing successes in a few places, accompanied by only sketchy 
descriptions of what was done or the difficulties of implementation. You 
will need to think very hard and reason very carefully about how you want 
to conceptualize (that is, define) the smart practice of interest and to assess 
the support requirements you think are most important. You need to do 
this even before you get to thinking about how the practice might work in 
the particular context(s) you have in mind (see the later discussion of this 
point under “But Will It Work Here?”). 


DESCRIBE GENERIC VULNERABILITIES 


It should be part of standard professional practice in describing smart 
practices to explain not only how and why they work but also how and 
why they fail, collapse, backfire, and generally make people sorry they 
ever tried them. That is, we should be told the nature of their generic 
vulnerabilities. A generic vulnerability is a potential weakness of the prac- 
tice that is somehow connected with its basic causal structure. It may 
have to do with a high sensitivity to small errors in execution, or with the 
environment in which the practice is being implemented (e.g., an envi- 
ronment that imposes certain insupportable stresses). 

Of course, all political and implementation environments are stressful 
to a certain degree, and we can reasonably include in the definition of a 
particular smart practice those features necessary to safeguard it against 
the more predictable and potentially damaging stresses. Without such 
safeguards, an otherwise smart practice can become a very dumb prac- 
tice. For instance, although privatizing certain municipal service func- 
tions is a smart practice when would-be private suppliers operate in a 
competitive market, it might become a very dumb practice under these 
circumstances: (1) if it were carried out in an environment monopolized by 
a single supplier; (2) if the bidding process were very corruptible, and cor- 
rupt interests were to discover this fact; (3) if inappropriate performance 
measures were stipulated in the contract; or (4) if the municipal contract 
management procedures were overly rigid or overly lax. To take another 
example, a high-expectations welfare-to-work program is vulnerable to 
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the condition of the local labor market: if unemployment is high and 
jobs are scarce, high expectations may produce in participants more 
defeatism about themselves and more cynicism about the “responsi- 
bility” that society is urging on them. A government/neighborhood 
partnership for park maintenance is, in a generic sense, vulnerable to, 
among other things, temptations on the part of policymakers to slowly 
shift more and more of the burden onto the neighbors while reallocating 
budgetary funds to other departments. 

Generic vulnerabilities are only the potential for trouble, it should be 
remembered. Whether the troubles materialize depends on the nature 
of the local environment in which the smart practice is implemented 
and on the success of various parties who are aware of the vulnerabili- 
ties in designing and implementing successful countermeasures. 
Contracting processes, for instance, can be designed to minimize cor- 
ruption, albeit at some cost. And neighbors entering into a partnership 
with the city regarding parks maintenance can insist on putting the 
terms of the partnership in writing and holding a well-publicized press 
conference to announce them. Even if such a document had no legal 
standing, it might give neighborhood representatives some useful 
political leverage in later years. 

Two particular types of vulnerability are especially worth attending to. 
One pertains to likely failures of general management capacity—such as 
a low general level of leadership talent or the lack of a “good government” 
ethos that would make it easier to implement this or any other practice 
successfully. The other pertains to weaknesses intrinsic to the particular 
practice itself—such as a service delivery program’s susceptibility to con- 
flict over whether to give priority to this or that catchment area or needy 
subpopulation, or a safety-oriented regulatory program’s inability to 
determine whether to err on the side of injury-tolerant leniency or 
costly stringency. 


BUT WILL IT WORK HERE? 


Assuming that you have sufficiently understood the essence of the 
generic smart practice, including its generic vulnerabilities, and have 
mapped the variety of supportive features that could increase its odds of 
success, in the end, you must still ask: “Assuming that this practice is indeed 
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smart in some contexts, is ours a context in which it can work well 
enough to warrant trying it?” Answering this question intelligently 
entails looking both at the source contexts, where the practice appears 
to have worked well, and at your own target context, where it is being 
considered for adoption. 


Assessing the Target Context 


Within your target context, a careful assessment of the present situation 
is in order, of course, but a static answer based on this assessment is not 
enough. You need to think also about what might be done at reasonable 
cost or risk to improve the prospects of the smart practice in the target 
context, were it to be implemented there. These actions fall into the fol- 
lowing two categories: 


e Safeguarding strategies. Consider the generic vulnerabilities of the 
smart practice: are the most dangerous of them likely to cause 
unacceptable trouble in your context? For instance, if excessive 
rigidity in the contract management process is a generic vulnerabil- 
ity of partnering with a nonprofit agency, are your contract man- 
agement institutions known to be unusually rigid? And if so, is 
there anything you can do to offset this problem? Might you, for 
instance, find someone in the contract management bureau who 
can serve as a special protector and expediter? Or, if you cannot do 
that, can you find some way to structure the contract terms so that 
the contractor is held accountable for achieving general results 
rather than for following specific procedures? 

e Enhancement strategies. Consider what I called earlier the “sup- 
portive features” that can help a practice to work better: What 
supportive features will be put into play? How well are they likely 
to perform? Can you do anything to improve them? For instance, 
can you attract top-notch personnel to manage this program or 
undertake this project? Can you obtain more stable funding than 
annual appropriations? Can you mobilize the press to take positive 
notice of what you are doing? Can you count on the support of key 
stakeholders and relevant political constituencies—or at least on 
weak action from opponents? 
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Evaluating the Source Contexts 


If you have to search very hard for smart practices that might be usable 
in your own situation, the chances are that such practices are not wide- 
spread. This means that the specimens you locate will come from juris- 
dictions, agencies, or locales where policymakers and administrators 
tend to look more favorably on novelty and innovation than is usually 
the case. Hence, their overall managerial capacity may be better than 
average—and perhaps better than the one in your own locale. 

If the source contexts are largely pilot or demonstration programs, you 
need to be particularly cautious, because (1) pilot program implementers 
often bring more enthusiasm and talent to bear on their work than the 
average program implementers, and enthusiasm and talent count for 
something; (2) the political and financial conditions at the pilot sites are 
probably more favorable (or less unfavorable) than those at the average 
site; and (3) bureaucratic resistance to a pilot program is typically less 
intense than to a permanent change that threatens existing values, status, 
job security, or work routines. 

How cautious should you be in extrapolating from successes observed 
in pilot or demonstration contexts? No systematic research exists to 
answer this question. However, a RAND Corporation analysis of a variety 
of juvenile crime prevention programs discounted the effectiveness levels 
attained in the pilot contexts by 15—40 percent when estimating a “scaling 
up penalty” that would apply when implementing the programs on a 
wide scale (Greenwood et al. 1995). Although the RAND analysts offered 
no explicit reasoning for choosing the penalty factors that they did, their 
choices do seem reasonable. 

If you are analyzing the possibility of implementing a smart practice not 
just in some known local context but on a wide scale, you should be con- 
cerned about more than the fact that pilot program results may be much 
better than average. You should also be concerned about the existence of 
many below-average sites where the smart practice would be implemented— 
some of them perhaps quite a bit below average. In an era when it was 
much less common than it is today to think about the federal devolution 
of program and policy responsibilities to state governments, federal 
policymakers—particularly political liberals—often worried about the 
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“Mississippi problem.” Mississippi was the rhetorical symbol of the poor, 
backward, and probably racist jurisdiction that would almost surely wreck 
or pervert any smart practice it was given responsibility for implementing. 


BACK TO THE EIGHTFOLD PATH 


Given the typical shortfall of good evidence relative to theory and specu- 
lation when it comes to assessing a smart practice, there is a danger of 
unwarranted optimism. Indeed, a common criticism of the best practices 
research tradition is that it becomes excessively enthusiastic about what 
appear to be good ideas before their worth is sufficiently tested.® 

But how much testing is “sufficient,” anyway? The answer has to be 
framed partly in terms of the costs of displacing what might actually be a 
better practice—perhaps even the practices currently in use (described 
earlier as “letting present trends continue”). However, if you are reason- 
ably confident that current practices are ineffective or harmful, the costs 
of wrongly abandoning them in favor of the new and untried may not be 
so high after all. Thus, although the new and untried should bear some 
burden of proof, it should not be an excessive one. The correct approach 
is to treat the risks and uncertainties involved in adopting some seem- 
ingly smart practice as comparable to the uncertainties associated with 
all the other alternatives under consideration. 

Of course, the costs of change—negotiation, insecurity, hard feelings, 
and so on—must also be counted against bringing in a new and seem- 
ingly smart practice. But such costs must be counted against any change, 
not just change to accommodate smart practice. Moreover, if institutions 
and people are very stuck in their ways, there may be benefits to change 
as such, not merely costs.” 


8. Unfortunately, excessive enthusiasm for experiments that eventually fail gives even 
appropriate enthusiasm for experimentation a bad name. 

9. Alternatively, if institutions and people are forever being reformed and reinvented 
and remodeled—as occurs in many public school systems—there may be benefits to 
stability, consistency, and focus. 


APPENDIX 


SPECIMEN OF A REAL-WORLD 
POLICY ANALYSIS 


The following text is excerpted from the policy brief “Mandatory Minimum 
Drug Sentences: Throwing Away the Key or the Taxpayers’ Money?” pre- 
pared by the RAND Corporation. It is provided as an example of a policy 
document and was chosen for its thorough and focused analysis as well as its 
concise presentation. My annotations appear in the numbered notes below the 
text. The notes of the authors of the brief are indicated by asterisks. 


PREFACE 


In response to public concern over disparity of sentencing by judges and 
brevity of terms served by criminals, state legislatures and the Congress 
have written into law minimum sentences for specific crimes. In this 
report, we estimate the cost-effectiveness of mandatory minimum sen- 
tences for crimes related to cocaine distribution. These estimates are 
made relative to the cost-effectiveness of spending additional resources 
on enforcement without mandatory minimums and on drug treatment.! 
Our central effectiveness measure is reduction of the nation’s cocaine 


Source: Jonathan P. Caulkins, C. Peter Rydell, William L. Schwabe, and James Chiesa, 
“Mandatory Minimum Drug Sentences: Throwing Away the Key or the Taxpayers’ 
Money?” (Santa Monica, Calif.: RAND Corporation, 1997). 

1. Cost-effectiveness analysis requires comparison. Often we compare the cost- 
effectiveness of some new state of the world with the old, existing one. But in this 
case the researchers compare several alternative new states of the world. 
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consumption, although we also examine reduction of cocaine-related 
crimes, along with decrease in cocaine spending, which is related to 
such crimes.” 

Because this report may be read by people with diverse interests, it is 
divided into two parts.° Readers interested principally in narcotics control 
and criminal-justice policy may wish to stop at the end of Part I. Part II has 
been prepared mainly for those also interested in the role and techniques 
of mathematical modeling in policy analysis, although some effort has 
been devoted to make it understandable to those not expert in this area. 

This research was supported by a gift from Richard B. Wolf of Richland 
Mills and by funding from The Ford Foundation. This study was carried 
out within RAND’s Drug Policy Research Center. The center’s work is 
supported by The Ford Foundation, other foundations, government agen- 
cies, corporations, and individuals. It carries out extensive assessments of 
drug problems at local and national levels. Those interested in further 
information should contact the center at RAND’s Santa Monica address. 


SUMMARY 


In recent decades, the American public has responded favorably to political 
leaders and candidates who have espoused longer sentences for the posses- 
sion and sale of drugs. Among the more popular sentencing extensions are 
“mandatory minimums,” which require that a judge impose a sentence of 
at least a specified length if certain criteria are met. For example, federal law 
requires that a person convicted of possessing half a kilogram or more of 
cocaine powder be sentenced to at least five years in prison. 

Mandatory minimums have enjoyed strong bipartisan support from 
elected representatives and presidential candidates.* To these proponents, 
the certainty and severity of mandatory minimums make them better 


2. These different “measures” are our “criteria.” Note that by making one “central” and 
two others obviously less so, the authors implicitly are differentially weighting the 
criteria. 

3. They are “telling the story” to two different audiences—which amounts to telling 
two different, though related, stories. Presumably the authors could have issued 
separate reports, but the chosen format keeps complementary analyses together. 

4. An important step in telling the story: establishing key features in the political 
environment. 
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able to achieve incarceration’s goals than are more flexible sentencing 
policies. Those goals include punishing the convicted and keeping them 
from committing more crimes for some period of time, as well as deter- 
ring others not in prison from committing similar crimes. Critics, how- 
ever, worry that mandatory minimums foreclose discretionary judgment 
where it may most be needed, and they fear mandatory minimums result 
in instances of unjust punishment.® 

These are all important considerations, but mandatory minimums asso- 
ciated with drug crimes may also be viewed as a means of achieving the 
nation’s drug control objectives. As such, how do they compare with other 
means? Do they contribute to the central objective—decreasing the nation’s 
drug consumption and related consequences’—at a cost that compares 
favorably with other approaches? In this report, we estimate how successful 
mandatory minimum sentences are, relative to other control strategies, at 
reducing drug consumption, drug-related crime, and the total flow of rev- 
enue through the cocaine market. The latter is a worthy objective in itself}— 
America would be better off if money spent on drugs were spent on almost 
anything else—and it is also associated with drug-related crime. 

We focus on cocaine, which many view as the most problematic drug 
in America today.® We take two approaches to mathematically modeling? 
the market for cocaine and arrive at the same basic conclusion: Mandatory 
minimum sentences are not justifiable on the basis of cost-effectiveness at 


5. The proponents are projecting outcomes. The authors of this report imply that 
they do not necessarily accept these projections and that this analysis will do its own. 
6. Proponents and critics clearly do not weight the evaluative criteria in the same 
way. The authors imply that they hardly even notice one another’s criteria. 

7. This description of the central objective of drug control policy establishes “drug 
consumption” as the critical numerator in the cost-effectiveness analysis, though 
others are to be considered as well. 

8. Narrowing the focus keeps the analysis manageable. The authors must justify the 
way they choose to narrow the focus, however. The shorthand version here is to call 
it “the most problematic drug,” though without specifying what that means. 

9. Not all models are mathematical. The phenomenon being analyzed here is com- 
plex, and formal modeling is both possible and desirable. Deciding on the values for 
particular parameters, such as the price elasticity of demand for cocaine, the cost to 
dealers of being caught, and the effectiveness of treatment, to name just a few, is 
somewhat speculative. The authors will use what evidence can be found in the 
research literature, however. 
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reducing cocaine consumption, cocaine expenditures, or drug-related crime." 


Mandatory minimums reduce cocaine consumption less per million tax- 
payer dollars spent than does spending the same amount on enforcement 
under the previous sentencing regime.!! And either type of incarceration 
approach reduces drug consumption less than does putting heavy users 
through treatment programs, per million dollars spent. Similar results are 
obtained if the objective is to reduce spending on cocaine or cocaine- 
related crime.!? A principal reason for these findings is the high cost of 
incarceration. (Note these findings are limited to relative cost-effective- 
ness. As mentioned above, mandatory minimums have been justified— 
and criticized—on other grounds.) 


Reducing Consumption: More Enforcement against Typical Dealers 


First, we estimate the cost-effectiveness of additional expenditures on 
enforcement against the average drug offender apprehended in the United 
States (whether that apprehension is by federal, state, or local authorities). 
In this approach, we track the flows of users among light-use, heavy-use, 
and no-use categories, and we analyze how overall cocaine market 
demand and supply respond to price. That is, if more money is spent on 
enforcement and incarceration, costs to dealers are increased, and so is the 
street price of cocaine; higher prices mean lower consumption.'? If more 
money is spent on treatment, consumption is reduced for most clients 
while they are in the program, and, for some, after they get out. We esti- 
mate the changes in total cocaine consumption over time for an addi- 
tional million dollars invested in the alternatives considered. These 
changes, discounted to present value, are shown in Figure A-1. 


10. The headline comes about as quickly as it possibly can. In italics, too. 

11. This restatement of the study’s primary conclusion establishes “million taxpayers’ 
dollars spent” as the critical denominator in the cost-effectiveness equation. 

12. No matter which criterion is chosen, it just so happens that treatment turns out 
to be a better investment than tougher incarceration procedures. 

13. Note the use of market theory to lay bare the heart of “the drug problem.” You do 
not often see a newspaper account of this subject that refers to price movements, 
demand, and supply. Yet the “laws” of supply and demand apply, in some sense, to 
drug markets as much as they do to the market for green beans. It is almost impos- 
sible to project the effects of alternative antidrug interventions without taking into 
account how the market mediates whatever is done. 
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Benefits of Alternative Cocaine Control Strategies 


Kilograms of consumption prevented 
per million 1992 dollars spent 
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The first two bars in the figure show the results of spending a mil- 
lion 1992 dollars* on additional enforcement by agencies at various 
levels of government against a representative sample of drug dealers. 
As shown by the first bar, if that money were used to extend to federal 
mandatory minimum lengths the sentences of dealers who would 
have been arrested anyway, U.S. cocaine consumption would be 
reduced by almost 13 kilograms. If, however, the money were used to 


* All cost calculations in this report are in 1992 dollars. The choice of a reference year 
for cost figures is arbitrary. We choose 1992 to facilitate comparison with the results 
of earlier analyses. To convert costs in 1992 dollars to costs in 1996 dollars (the latest 
year for which inflation data are available), multiply by 1.119. To convert kilograms 
of cocaine consumption reduced per million 1992 dollars spent to kilograms 
reduced per million 1996 dollars spent, divide by 1.119. 
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arrest, confiscate the assets of, prosecute, and incarcerate more dealers 
(for prison terms of conventional length), cocaine consumption 
would be reduced by over 27 kilograms.!4 Spending the million dollars 
treating heavy users would reduce cocaine consumption by a little over 
100 kilograms. 

Note we are estimating the impact of an additional million dollars. 
The results can be extrapolated to multiples thereof, but not to extremely 
large changes in spending. They certainly do not suggest that the most 
cost-effective approach is to shift all drug control resources from enforce- 
ment to treatment. Note also that we refer in the figure to “longer sen- 
tences” rather than to “mandatory minimums.”!» Data on drug dealers 
arrested at state and local levels are insufficient to isolate those associated 
with drug amounts sufficient to trigger mandatory minimums. Instead, 
we analyze a hypothetical policy of applying the mandatory minimum 
sanction—longer sentences—to all convicted dealers.'© 

The values shown are dependent, of course, on various assumptions 
we make. If the assumptions are changed, the values change. But for 
changes in assumptions over reasonable ranges, do the values change 
enough to make longer sentences more cost-effective than either of the 
other alternatives? We find they do not. 

As an example, the values shown are dependent on the time horizon 
in which one is interested. The reason for this is as follows. When faced 
with extended sentences, drug dealers will want more income today to 
compensate them for the risk of increased prison time. As a result, 
cocaine prices will go up and consumption will go down. Benefits from 


14. Note the importance of defining “the margin” where the policy operates. The 
policy is twice as effective if you can restrict “the margin” to dealers who would not 
have been arrested anyway. 

15. The political rhetoric might oversimplify by talking about “tougher” approaches 
to dealing, but the authors here distinguish two different forms of “tougher.” Even 
more forms exist, of course. 

16. All policies that deal with the future are “hypothetical.” Why do the authors go 
out of their way to use this term explicitly here? Probably because they wish to dis- 
tinguish the alternative they are analyzing from empirical specimens existing in the 
real world. They are more interested in a “generic” strategy based on mandatory 
minimum sanctions, a sort of composite of what does exist and what might exist in 
a more ideal form. 
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reduced consumption will thus accrue immediately, while the costs of 
the extended prison terms will stretch out into the future. In contrast, if 
more users are treated this year, the costs accrue immediately, while the 
benefits in terms of reduced consumption by those who stay off cocaine 
stretch out into the future. Figure A-1 takes account of these different 
allocations of costs and benefits across future years in that future costs 
and benefits are discounted annually, out to 15 years—a time horizon 
typical in analyzing public policy. Beyond that point, any further costs 
and benefits count as zero. What if that terminal point were moved 
closer? What if one had not just a discounted interest in anything 
beyond the immediate future, but no interest? If the time horizon is set 
early enough, the effect is to “zero out” both the future stream of costs 
from mandatory minimums and the future benefits from treatment. 
Figure A-2 shows the relative cost-effectiveness of the three programs 
analyzed when time horizons are set at various points, from 1 to 15 
years. At 15 years, the lines match the heights of the bars in Figure A-1. 
The time horizon must be reduced to only about three years before 
mandatory minimums look preferable to additional conventional 
enforcement, and close to two years before they look preferable to treat- 
ment. Hence, mandatory minimums appear cost-effective only to the 
highly myopic.!” 

We also analyzed the implications of changing other assumptions. 
For example, dealers would want to be compensated for the increased 
risk of imprisonment they would incur in the event of increased 
enforcement. But the typical person would demand less compensation 
for being imprisoned five years from now than next year, and we 
assume drug dealers are even more “present-oriented.” What would 
happen, though, if dealers wanted more risk compensation, and if they 
discounted future costs less heavily than we assume? Longer sentences 
would seem more burdensome than we assume, dealers would demand 
a higher premium for handling cocaine, the price of cocaine would rise 
even more with increased enforcement spending, and consumption 


17. A bit of wit and rhetoric! Who says you have to be dry and boring in telling your 
story? The authors can get away with it here because their result is extremely 
robust—and because they are well-known and respected analysts in this field. 
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| Favrea | Benefits of Alternative Cocaine Control Strategies for 
Different Time Horizons 


Kilograms of consumption prevented 
per million 1992 dollars spent 
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would fall even more. Consumption would also fall more than we 
expected if users were more responsive to price increases, i.e., if 
demand were more “elastic.” We attempted to swing the balance 
toward extended incarceration by simultaneously increasing risk com- 
pensation by one-third, cutting the dealer discount rate by two-thirds, 
and increasing the elasticity of demand by 50 percent.!* The general 
profile of our results did indeed change. The cost-effectiveness of lon- 
ger sentences tripled, while that of additional conventional enforce- 
ment doubled, and that of treatment rose by about a quarter. However, 
longer sentences remained the least cost-effective alternative, and 
treatment the most. 


18. This is another version of “sensitivity analysis.” 
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Reducing Consumption: More Enforcement 
against Higher-Level Dealers 


The first two bars in Figure A-1 represent enforcement approaches 
applied to a representative sample of all drug dealers arrested. Perhaps 
mandatory minimum sentences would be more cost-effective if they 
were restricted to somewhat higher-level dealers. By “higher-level deal- 
ers,’ we mean those who operate at higher levels of the drug distribu- 
tion system, who make more money and thus have more to lose from 
more intensive enforcement. To approximate such a restriction, we 
limit the set of offenders analyzed to those who are prosecuted at the 
federal level and possess enough drugs to trigger a federal mandatory 
minimum sentence. 

The results are shown in Figure A-3. There, the darkest bars represent 
the reduction in cocaine consumption from spending an additional mil- 
lion dollars in enforcement against the federal-level offenders just 
defined. The light bars are those from Figure A-1. Reading from the left, 
each light/dark pair of bars represents the same kind of program. The 
distribution of long sentences is the same for the first two bars, and the 
kinds of additional enforcement actions funded (arrest, seizure, prosecu- 
tion, and incarceration for conventional sentence lengths) are the same 
for the next two bars. 

As shown by the darker bars in Figure A-3, the consumption change 
achieved per million dollars spent on mandatory minimums is closer 
proportionately to that achieved through the other alternatives. While 
longer sentences for a representative set of all dealers have 46 percent of 
the effect of additional conventional enforcement against such dealers, 
federal mandatory minimums have 57 percent of the effect of additional 
conventional enforcement at the federal level.!? And, obviously, federal 
mandatory minimums do better relative to treating heavy users than do 
longer sentences for all dealers. To the higher-level dealers considered in 
this analysis, time in prison carries a greater cost, and amounts of cocaine 


19. These percentages are calculated from numbers to be read off Figure A-3. One 
might wish the authors had somehow warned the reader that these numbers cannot 
be read off what is actually drawn in A-3. 
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Benefits of Alternative Cocaine Control Strategies 


Kilograms of cocaine consumption 
reduced per million 1992 dollars spent 
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and other assets seized through increased enforcement are also larger. 
Thus, risk compensation must be higher, and the higher resulting cocaine 
prices drive down consumption more. Nonetheless, at any given level of 
government, or against any given type of dealer, mandatory minimums 
are less cost-effective than conventional enforcement. 

Why is that the case??? Drug enforcement comprises two types of 
components, each of which is costly for taxpayers and each of which 
contributes to keeping drugs expensive: (1) arrest and conviction, 
which impose costs on suppliers principally through the seizure of drugs 
and other assets, and (2) incarceration of convicted defendants. Amid 


20. Nice use of rhetorical question to vary the pace and tone. 


APPENDIX A 135 


complaints about the “revolving door” of justice, some overlook that 
arrest and conviction impose costs on dealers. In fact, on average, arrest 
and conviction impose greater costs on dealers per taxpayer dollar spent 
than does incarcerating dealers. Since mandatory minimums alter the 
mix of these two components of enforcement in favor of incarceration, 
they dilute or reduce the efficiency of enforcement relative to simply 
expanding both components proportionately. 

As with the light bars, the precise heights of the dark bars in Figure 
A-3 depend on various assumptions. Again, these include assumptions 
about such uncertain values as the compensation dealers would demand 
for increased imprisonment risk, the rate at which dealers discount 
future costs, the responsiveness of buyers to shifts in cocaine prices, 
what it costs to arrest a dealer, and the value of drugs and other assets 
seized. To test the sensitivity of our results to these assumptions, we vary 
the assumed values of factors such as these one at a time over substantial 
ranges. In all cases, conventional enforcement is more cost-effective 
than mandatory minimums, and treatment is more than twice as cost- 
effective as mandatory minimums. Even when assumed values are var- 
ied two at a time, large departures from assumed values are required for 
mandatory minimums to be the most cost-effective approach. In 
Figure A-4, for example, the government’s cost of arresting a dealer and 
the compensation a dealer wants for risking a year of imprisonment are 
varied simultaneously. The star shows the values assumed for the results 
in Figure A-3. As Figure A-4 shows, mandatory minimums would be the 
most cost-effective alternative only if arrest costs were to exceed $30,000 
and a dealer were to value his time at some $250,000 or more per year. 
Such dollar values would typify only those dealers at a fairly high level 
in the cocaine trade and who are unusually difficult to arrest.** For 
dealers costing less to arrest, cocaine control dollars would be better 


** Even for these dealers, it is possible that conventional enforcement would be more 
cost-effective than mandatory minimums. That would be the case if the range of 
conventional sentences could be matched to the range of offenders so that the 
highest-level dealers received very long sentences. 
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Which Program Is More Cost-Effective at Reducing 
Cocaine Consumption Under Different Assumptions? 
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spent on further conventional enforcement. For dealers demanding less 
risk compensation, the money would be better spent treating heavy 
users than on enforcement against such dealers.” 

Long sentences could thus be a smart strategy if selectively applied. 
Unfortunately, because mandatory minimum sentences are triggered by 


21. The uninitiated reader needs more help in interpreting Figure A-4 than the authors 
provide. The figure is divided into three regions. Each region contains combinations of 
values for the two parameters “cost to government” and “cost to dealers.” The boundar- 
ies between the regions indicate where the parameter combinations imply different 
winners in the cost-effectiveness competition that the authors have set up. Thus, man- 
datory minimum sentences win only when the parameter combinations fall in approx- 
imately the northeast quadrant, that is, when the costs are high both to dealers and the 
government. In other RAND publications, this sort of mapping is called “threshold 
analysis.” I call it, in Part I above, “break-even analysis.” 
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quantity possessed and because those thresholds are low, they are not 
selectively applied to high-level dealers. (Indeed, anecdotal evidence sug- 
gests that high-level dealers can sometimes avoid mandatory minimums 
more easily than their subordinates. High-level dealers have more knowl- 
edge about their organization to use as bargaining chips with prosecu- 
tors. Furthermore, such dealers often do not physically possess their 
drugs, as is required for a mandatory minimum to take effect; they hire 
others to incur that risk. To the extent that this occurs, mandatory mini- 
mum sentences would be even less effective than these results suggest.) 


Reducing Cocaine-Related Crime 


Of course, cocaine consumption is not the only measure of interest. 
Many Americans are worried about the crime associated with cocaine 
production, distribution, and use. Using data on the causes of drug- 
related crime and our cocaine market analysis, we quantify the approxi- 
mate crime reduction benefits of the various alternatives. We find no 
difference between conventional enforcement and mandatory mini- 
mums in relation to property crime; the former, however, should reduce 
crimes against persons by about 70 percent more than the latter. But 
treatment should reduce serious crimes (against persons as well as prop- 
erty) the most per million dollars spent—on the order of 15 times as 
much as the incarceration alternatives would. 

Why do we get these results??? Most drug-related crime is economi- 
cally motivated—for example, undertaken to procure money to support a 
habit or to settle scores between rival dealers. Fewer crimes are the direct 
result of drug consumption—crimes committed “under the influence.” 
However, we find very little difference between conventional enforcement 
and mandatory minimums in their effects on the money flowing through 
the market, and thus very little difference in their effects on economically 
motivated crime. We do find, as shown in Figure A-3, appreciable differ- 
ences in consumption effects, and thus appreciable differences in effects 
on crimes committed under the influence. The latter are more likely than 
are economically motivated crimes to be crimes against persons. 


22. Another well-placed rhetorical question. 
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Treatment, however, has an enormous advantage over enforcement in 
reducing the economic value of the cocaine market—larger even than 
that shown in Figure A-3 for reducing cocaine consumption. Why is 
that??? When a treated offender stays off drugs, that means less money 
flowing into the market. But when a dealer facing the risk of a longer 
sentence raises his price, say one percent, to compensate, buyers will 
reduce the amount of cocaine they purchase. The best evidence suggests 
that reduction will be something on the order of one percent.” Thus, the 
total revenue flowing through the cocaine market stays about the same, 
and so do the incentives for economically motivated drug-related crimes. 
Therefore, the effect of the enforcement alternatives is limited almost 
entirely to the relatively small number of crimes committed under the 
influence. Treatment, however, has an advantage against those crimes 
similar to that shown in Figure A-3 and an even greater advantage against 
the larger number of economically motivated crimes. 


Conclusion 


Long sentences for serious crimes have intuitive appeal. They respond 
to deeply held beliefs about punishment for evil actions, and in many 
cases they ensure that, by removing a criminal from the streets, further 
crimes that would have been committed will not be. But in the case of 
black-market crimes like drug dealing, a jailed supplier is often replaced 
by another supplier if demand remains. And not all agree whether 
mandatory minimums satisfy American standards of fairness and jus- 
tice. Even those who believe they do must ask themselves to what extent 
might it be desirable to give up some punishment of the guilty to gain 
some further reduction in cocaine consumption—consumption that 


23. And yet another! 

24. So they base their model’s parameter estimates on some sort of evidence after all! 
Because this report “Summary” is a summary of the logic of their analysis, the 
authors do not trouble the reader with the details of how these projections were 
made, that is, how their model is constructed and its parameters assigned numerical 
values. These will appear in the full report. The word evidence here perhaps is 
intended to reassure the reader that, although the analysis herein is “hypothetical,” as 
the authors indicated above, the hypotheses are to the extent possible based on a 
body of accumulated evidence. 
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can victimize the innocent.” This trade of punishment for drug use 
reduction must be considered because long sentences are expensive and 
cocaine control resources are limited. As we show, if reducing consump- 
tion or violence is the goal, more can be achieved by spending additional 
money arresting, prosecuting, and sentencing dealers to standard prison 
terms than by spending it sentencing (fewer) dealers to longer, manda- 
tory terms. (And that is to say nothing of what might be achieved by 
redirecting resources from enforcement to treatment—admittedly, a 
more difficult reallocation because those programs might be run by 
completely different agencies.) We find an exception in the case of the 
highest-level dealers—those who value their time most highly and are 
hardest to apprehend—where sentences of mandatory minimum length 
appear to be the most cost-effective approach. However, current manda- 
tory minimum laws are not focused on those dealers. 


25. The authors confront readers with a starkly defined trade-off. 
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THINGS GOVERNMENTS DO 


he following list of things governments do is meant to stimulate cre- 
ativity and give you ideas. The way to use it is to think about your 
policy problem and then go down the list, asking yourself: “Might there 
be any way to use this approach on this problem?” 
The “Why You Might Do It” discussion that accompanies each list of 
“What You Might Do” is necessarily brief. It is intended principally to be 
suggestive. 


I. TAXES 
A. What You Might Do 
1. Add a new tax 
2. Abolish an old tax 
. Change the tax rate 
. Change the tax base 
. Improve collection machinery 


AUAU 


. Tax an externality 


B. Why You Might Do It 


The most common conditions to which taxes are a solution are those in 
which there is inadequate government revenue for some purpose and— 
probably more important—those in which the structure of market prices 
fails to capture the true economic opportunity costs. If market prices are 
wrong, there are usually deeper structural reasons, such as oligopolistic 


141 
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power or government overregulation of some input, which might bear 
correcting by other means as well. 

Naturally, too many taxes can also be a problem, if they are inhibiting 
useful economic or social activity. 


Il. REGULATION 
A. What You Might Do 


. Add a new regulatory regime or abolish an old one 

. Write new standards or remove old ones 

. Tighten or loosen existing standards 

. Ban or prohibit something entirely 

. Improve the scientific and technical basis for writing standards 
. Close or open loopholes 

. Add, train, or better supervise enforcement personnel 


COND UT FB WN fF 


. Improve targeting of enforcement to catch bad apples, or to 
increase deterrence, or to increase resource efficiency 
9. Raise or lower the level of effective sanctions 
10. Tighten or loosen appeals procedures 
11. Change reporting and auditing procedures 
12. Add, subtract, or improve complaint mechanisms for workers or 
the public 


B. Why You Might Do It 


Distinguish three quite different types of regulation. One aims at prices 
and outputs in natural monopolies, for instance, the historical regulation 
of local telephone service by a public utilities commission. As this exam- 
ple suggests, technological change (e.g., cell phones, broadband) can 
undermine natural-monopoly production and render this form of over- 
sight irrelevant. 

A second type—sometimes called “social regulation’—is common in 
regard to health and safety issues. It aims to correct imperfections arising 
from poor market information or from excessive frictions resulting from 
the use of civil law (usually tort or contract) remedies. Drug safety regu- 
lation by the FDA is an example. Bank solvency regulation also fits this 
category. Two sorts of problems are common in this type of regulation: 
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too little regulation and too much. Scientific uncertainties, technical dif- 
ficulties of measurement, and political pressures typically lead to both of 
these problems under varying conditions. 

A third type of regulation concerns entry, exit, output, price, and 
service levels in supposedly oligopolistic industries (e.g., transportation). 
Administering this type of regulation presents large problems of collect- 
ing information and of coordinating the outputs of many firms. Politically, 
there are often problems of anticompetitive “capture.” The deregulation 
movement that has gathered political momentum since around 1978 has 
led to a new appreciation of how much beneficial competition there 
might be in these industries if government were simply to let go. 

Most air and water pollution regulation is thought of as social regula- 
tion. However, administratively (and sometimes politically), it is more 
like the third type of regulation, inasmuch as the principal laws now on 
the books involve government agencies in coordinating the outputs of a 
variety of firms. 


Ill. SUBSIDIES AND GRANTS 
A. What You Might Do 


1. Add a new one 

2. Abolish an old one 

3. Change the level 

4. Change the marginal rate 

5. Introduce, abolish, or change a formula by which subsidies are 
allocated 

6. Modify the conditions of receipt or eligibility 

7. Loosen enforcement 

8. Tighten enforcement 


B. Why You Might Do It 


Incentive effects. Subsidies and grants are often used to stimulate activ- 
ities that neither markets nor nonprofit or voluntary action appears to 
produce in adequate quantity or quality. They also play important roles in 
the system of intergovernmental relationships—when one level of gov- 
ernment wishes to encourage another level of government to do certain 
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things—and in the system of relationships between governments and 
nonprofit organizations. 


Wealth effects. Grants and subsidies also transfer resources to people 
or organizations or levels of government in order to make the recipients 
wealthier. 


Some design problems. It often happens that you want to create incen- 
tive effects but not wealth effects, or vice versa. For instance, you may 
wish to make poor people wealthier via grants and subsidies but without 
diminishing work incentives. Or you may wish to encourage businesses 
or universities to undertake more research and development of a certain 
kind but without unduly enriching them or allowing them to use the 
subsidies inefficiently. 

Note that subsidies and grants are typically administered with various 
guidelines or conditions attached. The threat to remove a long-time 
grant or subsidy for violation of the guidelines or conditions can act as a 
type of regulatory sanction, thus making certain grants and subsidies 
into a peculiar regulatory hybrid. 


IV. SERVICE PROVISION 


A. What You Might Do 


1. Add a new service 

. Expand an existing service 

. Organize outreach to potential beneficiaries not now using the service 
. Better customize an existing service to a particular subpopulation 
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. Provide vouchers for a particular service so that people may choose 
from an array of competitive service providers 
6. Link two or more existing service delivery systems to take advantage 
of potential synergies or to make life easier for service recipients 
7. Reduce service users’ difficulties in accessing the service by 
a. going online 
. computerizing intake and eligibility processes 
simplifying forms 
. colocating services 
permitting appointments by phone 


omang 
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f. facilitating personal inquiries and complaints 
g. improving payment options 


B. Why You Might Do It 


Services come in two basic flavors. Desired services are those that people 
want, such as parks and good schools. Paternalistic services are those that 
people may or may not want but that outsiders want them to have because 
there is some potential payoff to the outsiders (e.g., rehabilitative services 
for the mentally ill, organized shelters for the homeless, job search ser- 
vices for individuals on welfare). It is a lot easier to design a service provi- 
sion system for desired services than to do so for paternalistic services. 


V. AGENCY BUDGETS 
A. What You Might Do 


1. Add a lot to the budget 

. Add just a little to the budget 

. Hold the budget at last year’s level 

. Cut the budget a little 

. Cut the budget a lot—to the point of beginning to terminate the 
agency 

6. Shift allocations from one budget item to another 
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B. Why You Might Do It 


You may want to adjust an agency’s budget according to whether you like 
what it does. In addition, how you manipulate an agency’s budget sends 
political signals about the degree of satisfaction or dissatisfaction with 
the agency’s performance and so may be thought to have incentive effects 
as well as wealth effects. It is not easy to use the budget as a means of 
creating incentive effects, however. 


VI. INFORMATION 
A. What You Might Do 


1. Require disclosure 
2. Direct government rating or certification 
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3. Standardize display or format 

4. Simplify information 

5. Subsidize production of information 

6. Subsidize dissemination of information 


B. Why You Might Do It 


Information production, dissemination, and validation may be subopti- 
mal due to the declining average (and sometimes marginal) cost nature 
of the activity. Information consumption may be suboptimal due to the 
hidden costs of consumption (such as time spent reading or hearing or 
interpreting or sifting or verifying). 


VII. THE STRUCTURE OF PRIVATE RIGHTS 
A. What You Might Modify or Create 
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. Contract rights and duties 

. Property rights 

. Liability duties 

. Family law 

. Constitutional rights 

. Labor law 

. Corporate law 

. Criminal law 

. Dispute-resolving institutions other than litigation and courts 
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B. Why You Might Do It 


In recent years, two of the biggest issues drawing the attention of policy 
analysts and economists interested in legal institutions are the economi- 
cally efficient incidence of risk—it should fall on the party that can man- 
age it at the lowest social cost—and the costs involved in administering 
any adjudicative system. Since private-law duties and rights do a lot to 
allocate risk (e.g., if your product exposes the user to risk and ultimately 
injury, you may be liable for damages, unless perhaps the user abused or 
misused it or agreed to assume the risks of use), adjusting laws is some- 
times a powerful policy intervention mechanism. Also, much creative 
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thinking has gone into finding ways to reduce the administrative and 
adjudicative costs. 

In addition to these economic matters, there is also concern about 
compensation for harm. Laws can be changed so as to shift wealth—in 
some prospective, actuarial sense or in a real, present-time sense— 
among different interests or classes of people. 

The wealth-shifting and risk-shifting effects of legal changes may both 
work in the desired direction, or they may work at cross-purposes. In addi- 
tion, both may work together with, or at cross-purposes with, the desire to 
reduce administrative and adjudicative costs. 


VIII. THE FRAMEWORK OF ECONOMIC ACTIVITY 
A. What You Might Do 


= 


. Encourage competition 

. Encourage concentration 

. Control prices and wages (and profits) 

. Decontrol prices and wages (and profits) 
. Control output levels 

. Decontrol output levels 

. Change tax incentives up or down 

. Provide public jobs 

. Abolish public jobs 
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B. Why You Might Do It 


Supporting more government intervention. On the supply side, there 
may be monopoly or oligopoly problems. On the demand side, consum- 
ers may be relatively nonmobile or otherwise vulnerable to exploitation— 
and the same may be true of workers. 


Supporting less government intervention. You may decide that politi- 
cal forces have captured the government administrative apparatus and 
perverted the intent, or that the information costs to government 
entailed in doing the job well are simply too high, or that technology has 
changed and made an older form of government intervention less appro- 
priate or effective or efficient. 
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IX. EDUCATION AND CONSULTATION 
A. What You Might Do 


1. 
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Warn of hazards or dangers 


. Raise consciousness through exhortation or inspiration 

. Provide technical assistance 

. Upgrade skills and competencies 

. Change values 

. Professionalize the providers of a service through training or certi- 


fication or licensing 


B. Why You Might Do It 


People may be unaware of a problem or an opportunity. They may be 


careless or unfeeling. There may be too many untrained or unskilled 


people in jobs demanding too much responsibility. 


X. FINANCING AND CONTRACTING 
A. What You Might Do 
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. Create a new (governmental) market 

. Abolish an existing (governmental) market 

. Alter reimbursement rates 

. Change the basis for reimbursement (e.g., cost-plus, price per 


unit, sliding scale dependent on quantity, performance bonuses 
or penalties) 


. Lease governmentally held resources 

. Alter user fee structure 

. Redesign bidding systems 

. Change contract enforcement methods 
. Furnish loans 

. Guarantee loans 

. Subsidize loans 

. Set up a public enterprise 

. Dismantle a public enterprise 

. “Privatize” a hitherto public enterprise 
. Modify insurance arrangements 

. Change procurement practices 
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B. Why You Might Do It 


Capital and/or insurance markets may be working inefficiently. The gov- 
ernmental contracting and procurement machinery may not be operating 
well—it may be too rigid, or too corrupt, or too expensive, or too slow. 


XI. BUREAUCRATIC AND POLITICAL REFORMS 
A. What You Might Do 


The number of possibilities is too great to list. It ranges across such 
activities as reorganizations, replacing top supervisory personnel, 
improving information systems, and raising wages and salaries. 


B. Why You Might Do It 


The substantive reasons are too numerous to list. We should note, though, 
that in many policy contexts there are important political and symbolic 
considerations for undertaking bureaucratic and political reforms. The 
political considerations often involve enhancing the power of one social 
interest or point of view at the expense of another. The symbolic consid- 
erations often involve ducking the really hard or impossible problems at 
the social level in favor of doing something readily seen in a domain over 
which government appears to have control (that is, its own operations). 


APPENDIX 


UNDERSTANDING PUBLIC AND 
NONPROFIT INSTITUTIONS: ASKING 
THE RIGHT QUESTIONS 


P olicy analysis, properly done, requires you to think not only about the 
technical aspects of governmental action but also about its institu- 
tional aspects—that is, the institutions that implement policy. Whether 
implementation goes well or poorly depends in part on whether the rele- 
vant institutions want to facilitate or impede the policy at hand. But moti- 
vation is not the whole story; capacity is at least as important. Here I want 
to draw attention primarily to the aspects of organizational structure and 
process that bear on capacity, and I do so not by offering a detailed exposi- 
tion but by posing some (41) questions that the analyst ought to be asking. 


MISSION 


1: What is the mission of the agency? 


e As expressed in authoritative sources? 
e As understood and enacted by agency managers and employees? 


ENVIRONMENT 
2: What support/opposition does the agency have for its mission, and 
for itself, in its “authorizing environment”—that is, the totality of 
actors whose legal and nonlegal attitudes and actions determine 
agency legitimacy in the polity? 
e Evidence from budgetary allocations? 
e Other evidence? 
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3: Is the task environment relatively placid, changing predictably, or 
changing unpredictably? Examples of what is meant by these terms: 


e Placid: telephone company in 1975 

e Changing predictably: hospitals in the 1970s, before managed care 
but in the era when changing technologies were introduced and 
when overcapacity was looming 

e Changing unpredictably. telecommunications companies today; 
hospitals today, facing changes in the insurance marketplace, in 
technology, and in government policies 


Note that the less predictability and the faster the pace of change, the 
greater the need for getting information from the organization’s field 
people rapidly; allowing them to make at least some decisions on their 
own; and making grand strategic shifts from the center. 


4: Who are the agency’s main competitors for resources and/or domain 
of legitimate action? 

5: Does the agency have a comparative advantage—or disadvantage— 
in meeting the competition? 

6: Does the agency face rivals who don’t merely compete but are down- 
right hostile to aspects of the agency’s mission or philosophy? 


PERFORMANCE MEASUREMENT 


7: What metrics are available to tell us how effectively and efficiently 
the agency is performing? 


e Producing outputs and outcomes? 


Outputs are what you can operationally count as a result of the agency’s 
productive work, such as number of students graduated, number of acres 
of forest thinned, number of passengers carried per day. Outcomes are 
consequences of the outputs that we actually value per se, such as greater 
employability of those students, greater fire resistance of those forests, 
reduction in transportation time or in pollution levels or increases in 
transportation satisfaction. 

It is usually (much) harder to measure outcomes than outputs. 
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Outcomes are the product of more than the agency’s outputs. Hence 
crediting or blaming the agency for the quality of the outcomes is not 
straightforward. 


8: Compared to what benchmarks or standards? 


e Other similar organizations? 
e The same agency in previous years? 
e An absolute standard? 


9: What additional metrics would you, ideally, like to see? 
10: Who are the agency’s “customers,” if any? Are they being well served? 
What is their opinion? 


e Itis easy to see a park user as a “customer” of the parks department. 
e But is a taxpayer a customer of the IRS? 
e Is the student a customer of the school? Or is her family? 


TECHNOLOGY 


Technology is a slightly fuzzy word, more easily defined by example than 
by abstractions. 


11: To what extent does the agency use a service-delivery technology? A 
regulatory technology? A people-changing technology? A project 
technology? 


e Service-delivery technology: a transportation agency’s provision of 
vehicles and other facilities for use by patrons; writing and mail- 
ing Social Security checks; issuance of annual auto registration 
plates or stickers 

e Regulatory technology: command-and-control activities, such as 
environmental inspection and enforcement; OSHA regulations; 
restaurant sanitation inspections 

e People-changing technology: education; probation; child protective 
services 

e Project technology: issuance of a land-use plan; construction of a 
convention center; immunization of all youngsters against this 
year’s strain of flu 
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12: Is it a strong or a weak technology? 


e A strong technology—such as writing Social Security checks—is 
replicable and works in all contexts. 

e A weak technology—such as counseling probationers—is not eas- 
ily replicated and is sensitive to social context and individual- 
level competency. 


The structure and process you use to run an agency will depend to 
some degree on the type of technology you are implementing, partly for 
technical reasons and partly for political reasons: you get into trouble if 
you adopt a regulatory attitude when you should be serving a “customer,” 
and vice versa. 


13: Is the agency’s authority structure appropriate to the agency’s mis- 
sion, technology, and human resources? 

14: Is the agency’s internal division of labor appropriate to the agency’s 
mission, technology, and human resources? 

15: Does the agency have effective means—formal and/or informal—to 
create working relationships across unit boundaries? 


PRODUCTION/DELIVERY PROCESSES 


16: What is (are) the principal process(es) the organization uses to imple- 
ment its production technology(ies)—that is, to produce its outputs? 


This question is complicated because it focuses on processes that con- 
nect the organization’s activities, the human and physical materials it 
works on, and the way in which it transforms materials. 


e In environmental and other kinds of command-and-control regu- 
lation, we have processes of “standard setting, inspections for com- 
pliance with standards, and the threat and actuality of punishment 
for noncompliance,” a complicated form of deterrence. 

e Ina welfare program, the process involves determinations of eligi- 
bility and amount. 
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FRONT-LINE WORKERS AND CO-PRODUCERS 


17: Are the front-line workers doing a good job? Are they competent? 


18: 


19: 


20: 


21: 


Motivated? Adequately supported by the organization? 

What systems of incentives, supervision, and support are in place to 
motivate them, help them, and hold them accountable? Are the 
means of recruiting them suited to getting individuals with the right 
qualifications? 

Does the organization make full use of the talents and information 
residing at the front-line level? 

To what extent does the agency also rely on “co-producers”—people 
and organizations that share in the production process without 
being employed by the agency? 


e Welfare recipients are co-producers, along with the welfare agen- 
cy, to the extent that they assist, and are assisted by, the agency in 
the recipients’ efforts to find jobs. 

e Parents co-produce their children’s education along with the chil- 
dren’s teachers. 

e Along with the environmental regulatory agency, complying firms 
co-produce environmental improvement. 


Does the agency manage its co-producers well? Do they provide the 
proper tools? Information? Motivation? 


PARTNERS AND OTHER OUTSIDERS 


22: 


23: 


24: 


To what extent does the agency rely on public and not-for-profit 
partners to accomplish its mission? Does the agency manage these 
relationships effectively?! 

Does it rely heavily on for-profit vendors? Does it manage these rela- 
tionships effectively? 

Does it effectively manage its relationships with governmental “over- 
head” (or “control” or “staff”) agencies—such as the Departments of 
Finance or Budget, Personnel, Procurement, Audit, and so on? 


1. For an extensive discussion, see Bardach (1998). 
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CENTRALIZATION/DECENTRALIZATION 


25: If the organization delivers services at multiple sites (as in a school 
district), does it have an effective balance of centralization and 
decentralization? 

26: Does it use its internal budget-making procedures to structure incen- 
tives for performance improvements and cost reductions? 


CULTURE AND COMMUNICATIONS 


27: Does the agency have a relatively “strong” or “weak” culture? 


e Strong culture: the U.S. Navy; the local fire department; the Forest 
Service; the California Legislative Analyst’s Office 

e Weak culture: many contemporary public health departments; 
many inner-city public schools 


28: Is the culture relatively hierarchical or egalitarian? Rules-oriented or 
performance-oriented, or both? Where do efficiency and cost mini- 
mization fit in its culture? 

29: Are communications within the agency relatively unconstrained by 
hierarchy and/or by subunit boundaries? Are people relatively unafraid 
to speak their minds, or are they circumspect and cautious? Is entre- 
preneurship encouraged or discouraged? 

30: Do senior managers attempt to “lead through culture”? How? 

31: Are creativity and innovation valued within the agency? Does the 
agency make systematic efforts to stimulate creativity and innovation? 


POLITICS 


32: Are there factions within the organization? If so, are they based on 
professional, ideological, or bureaucratic cleavages? On other factors? 

33: Does factional competition or conflict degrade organizational per- 
formance or—by stimulating effort and healthy competition— 
improve it? 

34: How do senior managers deal with the existence of factions? 


LEADERSHIP 


35: Who, if anyone, is a leader in this organization? 
36: How does such a person gain and preserve legitimacy? 
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37: How effective is (are) the leader(s)? 
38: What functions do they play in the organization? 
39: What strategies do they use to carry out their leadership functions? 


CHANGE 


40: Is there a culture of continuous improvement—a term originally asso- 
ciated with the Total Quality Management (TQM) movement—and 
learning from mistakes? 

41: Does the agency have the capacity and motivation to scan the envi- 
ronment for signs of opportunity or danger? If such signs are pres- 
ent, can the agency adapt effectively? 


APPENDIX 


STRATEGIC ADVICE ON THE DYNAMICS 
OF GATHERING POLITICAL SUPPORT 


his Practical Guide is largely about the intellectual aspects of policy 

analysis done in the public interest. Its lofty focus should not obscure 
the fact that the adoption of policy occurs through a process that is often 
untainted by much intellectuality at all and is subject to the pushing and 
hauling of many sorts of stakeholders, not all of them concerned with 
“the public interest.” 

All policy is political, whether the politics takes place in a back room 
or within a legislature, an organization, or a community. You must build 
support and neutralize opponents. To do this, at the interpersonal level, 
you must make arguments, frame and reframe “the facts,” call in favors, 
imply threats. At the organizational and institutional levels, you must 
mobilize allies, manipulate arenas and calendars, offer (and extract) con- 
cessions, and negotiate side-payments. 

All these issues of attitude and influence involve a good deal of calcula- 
tion and estimation: which agency’s budget will grow larger; whose turf 
will be diminished? How much weight, or influence, do these supporting 
and opposing interests carry with the key decision makers anyway? I ignore 
all such questions here, although in your work you must both ask them 
and develop the information to answer them. The elementary answers are 
fairly obvious, however, and the sophisticated answers would take up too 
much space. (Example question: will agencies support policies that expand 
their budgets and powers? Elementary answer: yes. Sophisticated answer: 
usually yes, but probably not if such expansion risks changing internal 
power relations drastically or exposing the agency to possible failure and 
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criticism, etc. Even more sophisticated answer: ... [this I leave to your 
imagination and experience].) 

One might say that these matters of attitude and influence are instanc- 
es of static analysis. Journalists’ and political scientists’ accounts of the 
legislative process are rich in such instances. I limit my discussion here, 
though, to dynamic analysis of the process, which involves questions of 
sequence (what you should do before, or after, you do some other thing) 
and timing (what to do “early” or “late” in the projected course of a 
months- or possibly years-long campaign).! 

The assumption here is that one can make a list of generic “things to 
do” based on general knowledge of how such campaigns play themselves 
out, then postulate a generic flow of threats and opportunities over the 
course of such a campaign, and, finally, map the list of “things to do” into 
this flow with the object of maximizing the chances of winning. More 
precisely, you want to do the mapping not only with winning in mind but 
also with the idea of conserving political and other resources so that they 
will be available for other campaigns. 

As a basic example, let us consider the case of a political entrepreneur 
trying to get a bill passed by a legislative body. The general outline of the 
process and of entrepreneurial strategy should apply to roughly analo- 
gous situations as well, such as trying to win approval from senior man- 
agement to try some organizational innovation. 

Start with the idea that you need to accumulate Support? from a vari- 
ety of Interests (construed very broadly, and including individuals as well 
as groups or organizations) up to the point that you have Enough Support 
to swing the balance toward legislative victory for some Proposal. For the 
sake of concreteness, think about passing a bill to remove an existing 
statute requiring mandatory minimum sentences for drug users and 


1. For a more elaborate discussion of dynamic analysis in general, see Bardach 
(2006); and also a very early effort, Bardach (1972). For an effort to describe the 
dynamics of building an interagency collaborative, see Bardach (2008). 

2. From here on, I capitalize terms—and italicize them on first occurrence—that 
could be integrated into a formal model and, in that context, would deserve precise 
definition. There are relatively few such terms, and they would make an economical 
but, I believe, powerful model. For our purposes here, however, the context and the 
everyday meanings of the terms make the concepts clear enough. 
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low-level dealers, while retaining them for high-level dealers, as would be 
implied by the RAND analysis in Appendix A. 

Assume that there is a hazily perceived and somewhat movable End- 
date, months or years off and linked to Allies? willingness to plug along 
before concluding that the cause is lost and that other drug policy issues 
now deserve their attention and other Resources. The basic rule is this: if 
your Proposal accumulates Enough Support at the moment of the End- 
date, you win; otherwise you lose. What sequencing and timing strategies 
and tactics should you deploy? 


Sequencing 


First consider the problem of sequencing, in particular whether getting 
some Interests on board first will make it politically easier to approach 
others subsequently. Who might these primary Interests be? 


e Persons with drug-policy expertise who agree with the RAND analy- 
sis, as well as persons or groups with a reputation for soundness, judg- 
ment, and ideological compatibility, might clear the way for Interests 
concerned about the practicality and efficacy of your Proposal. 

e For Interests with an internal hierarchy, such as a state substance 
abuse agency or a political party organization, getting Support from 
experts or influentials lower down might require getting clearance 
from individuals higher up in the hierarchy. 

e Access to particularly Weighty individuals, such as the chair of the 
Assembly Committee on Criminal Justice, might require working 
through a chain of introductions and connections. 


Timing 

One aspect of timing involves trying to stimulate bandwagon effects: get 
some Weighty Interests on board early in order to create the impression that 
success is inevitable, which will (1) draw in Interests concerned primarily 


with being on the winning side and, (2) Neutralize, at least for the time 
being, Interests who might otherwise go into active opposition. 


3. Allies are Supporters who are also active in trying to woo potential new Supporters 
and in attempting to Neutralize actual or potential Opponents. 
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Other aspects of timing are more subtle. Suppose that you have a 
rough but generally reliable time-profile (or map) of Opportunities and 
threats that tend to emerge relatively early or relatively late in the course 
of a campaign. You are thus able to time your actions so as to capitalize 
on the Opportunities and Neutralize the threats. You are also able to 
(roughly) plan to expend certain Resources early in order to capitalize 
on the Opportunities that come early on and then vanish forever, and 
to conserve certain (Depletable) Resources for the threats that you can 
anticipate toward the middle and end of the process.’ And, if you are 
lucky, you may never have to expend those Resources at all. 

For instance, early in the campaign you want to win over those 
Interests who will be enthusiastic Supporters and active recruiters—that 
is, real Allies. They will help you with raising money and gaining access 
to other such tangible Resources as office space, talented staffers, strategic 
advisers, and communications experts. 

Unfortunately for you, repealing mandatory minimum sentences is 
not a policy Proposal that engages a natural constituency of Resource- 
rich potential Allies. You might find a few drug-policy and criminal- 
justice policy experts, some grassroots ethnic advocacy groups (who 
observe that their ethnic confreres are disproportionately locked up under 
the current laws), possibly the state university system (which sees scarce 
budget dollars diverted from higher education to build more prisons for 
ever-growing populations of prisoners), and perhaps the odd taxpayer 
group concerned with the budget and the tax burden more generally. 
But, whatever the limitations of their endowments, these are the sorts of 
Allies with Marketing Resources that you will need to go into business. In 
fact, you need them so much in the early days of the campaign that you 
will need to expend cash, call in political Debts, and ask for political 
Favors (that is, go into Debt yourself). 

These early days represent a significant Opportunity for utilization of 
your scarce Resources for one very important reason: the Opposition will 
not yet have mobilized. (The Opposition is likely to include, among 


4. As a logical matter, you would also be able to plan to Neutralize any threats that 
emerged early and seize Opportunities that emerged late. But as an empirical matter, 
I do not think these possibilities important enough to discuss here. 
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others, politicians who find demagogy on “law and order” a useful elec- 
toral tool; contractors and subcontractors who profit from the prison- 
building boom; rural communities for whom prisons are a valuable 
source of employment; the prison guards union.) You and your Allies will 
be able to hold the stage yourselves, make your pitch—that is, Tell Your 
Story—and hype the bandwagon aspects of your coalition’s progress. 

Why is the Opposition silent? It takes time for its members to organize 
themselves, and some may be temporarily occupied with more pressing 
matters anyway. Most important, they know that most such attempts at 
significant policy change simply die of their own accord, and so they may 
not need to go out of their way to kill this one. Indifference, skepticism, 
the comfort of the known, and the veto-enabling character of our politi- 
cal institutions (e.g., checks and balances) pose a high barrier to change. 
Although these Opposition skeptics are right, they nevertheless are giv- 
ing you a brief moment in the sun, which you would be wise to seize, 
even if it means taking certain risks. 

As the Support for your Proposal visibly rises, however, the Opposition 
manages to organize a countermobilization. It succeeds in stalling your 
efforts to persuade Interests that are still uncommitted and even man- 
ages to Neutralize a few of the Interests you thought you had won to 
your cause. 

Now is the time to expend some of the Depletable Resources” you have 
been conserving and perhaps to borrow a little against the future as well. 
What are these Resources? 


e First, Debts—owed to you (and your coalition Allies) in exchange 
for past Favors you have done the indebted parties—that you can 
now call in. 

e Second, policy Concessions you can make—that is, amendments to 
your Proposal. In this case, you might offer to enlarge the definition 
of “high-level dealer” beyond what you had originally proposed. 


5. A Depletable Resource is one that, when used, is used up. Examples are money and 
Debts that are called in. Requests for Favors establish future liabilities; and the capac- 
ity for honoring them is in some sense Depletable, the exceptional case being that of 
reinvestment of Favors in such a way that overall capacity is augmented, much as 
business loans can lead to larger profits and even more successful enterprise. 
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e Third, normative expectations about the contributions of Allies 
that you can implicitly enforce—for instance, “as we are all in this 
together, it would be helpful if one of your people could make a call 
on Representative X.” 

e Finally, Debts that you can create (up to some limit, of course, 
which is why this is a Depletable Resource) by asking Favors. 


But note well: although expending all such Resources is costly, 
Concessions may be doubly costly, because a Concession on policy usu- 
ally represents either a goal or an objective foregone and also, possibly, 
a seeming breach of loyalty with Allies who have already signed on in 
the hopes of passing the very element of the Proposal that you are giv- 
ing up. (Perhaps the ethnic advocacy Interests who signed on early will 
be severely alienated by your enlargement of the pool of “high-level 
dealers.”) Hence, in this recovery phase, you stand a very good chance 
of raising fears among all your Allies that their hearts’ desires are also 
in jeopardy. 

Given that you can anticipate these coalition-fragmenting develop- 
ments, you can take one or more prophylactic measures: build in potential 
Concessions (bargaining chips) of relatively low value to you and your 
Allies early in the contest—such as starting with an extremely restricted 
definition of “high-level dealers’—so that they may be ceded during the 
recovery phase; accumulate a surplus of Allies and Supporters as a buffer 
against defections by the angry and disappointed during the recovery 
phase; develop a reputation for integrity that can withstand suspicions of 
carelessness and disloyalty. 

At some point, the Proposal is embodied in a bill, and the contest over 
the bill gets structured by the procedures of the legislature—its commit- 
tees, its rules, and its calendars. The Attentive Public expands as well. 
Earlier, it had included mostly Interests concerned about the policies 
and Resources pertaining to drugs and criminal justice. Now it includes 
legislators (and staff) who are concerned about the Proposal’s possible 
uses in partisan, institutional, and interpersonal competition. The rele- 
vance of Marketing Resources declines relative to that of more political 
Resources, which can be brought to bear on the manipulation of legisla- 
tive procedures and the augmentation of Support or the Neutralizing of 
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Opposition by the use of Side-payments. Neutralization is especially use- 
ful—for instance, promising to support the prison guards’ union in its 
next bid for a wage increase. 

Also, as the contest approaches its increasingly less obscure End-date, 
and the margin of possible victory or defeat becomes thinner and clearer 
(in contests destined to be close), both Allies and Opponents engage in 
riskier maneuvers in regard to an increasingly heterogeneous and volatile 
Attentive Public, with somewhat unpredictable effects. In this End-game 
phase of high risk and uncertainty, you want to have a goodly store of 
highly Fungible Resources, such as callable Debts and Good Will, which 
also happen to be Depletable. Hence, it is prudent to conserve these more 
political and Fungible Resources until this time, even though they might 
have been of some use in earlier phases as well. To put it epigrammati- 
cally, if the Speaker of the Assembly owes you a Favor, it’s the ace of 
trumps, and don’t play it until you absolutely need to. 
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